Judging GCSE English: Efficiency and Reliability
We have already established that schools can judge reliably, in line with the mark scheme, and we can derive reasonable grades for pupils. At what cost, however? If the effort involved is too great then the gains are likely to be overshadowed by any increase in workload.
For Comparative Judgement there is always a trade off between the number of judgements you make and the reliability of your results. The more judgements you make, the more you can trust in your results. Our rule of thumb to estimate the number of judgements you need to make to achieve high reliability is to multiply the number of scripts you have by 10. So if you have 120 pupils you need to make 1,200 judgements. The 1,200 judgements are then divided by the number of teachers you have available. If you have 12 teachers they would need to do 100 each. As you see two scripts every time you make a judgement, that means that every script is judged around 20 times.
There are three critical numbers in this equation; the number of judgements per pupil; the pupil to teacher ratio; the length of time it takes to make a judgement. Reducing any of these numbers reduces the time judging takes. So how did our schools fare?
How long does a judgement take?
The median judgement time was 23 seconds for the writing essay and 22 seconds for the reading essay.
How long does it take to get a reliable mark for a pupil?
We can use the data we collected to see how accurate the judging was after different numbers of judgements.
The figure below shows the error in the marks after different lengths of time per judge. After 110 seconds per judge (around 5 judgements for a judge, 10 judgements per script) the marks are accurate to within +/- 1.8 marks for the reading essay and +/- 3.2 marks for the writing essay.
After 160 seconds per judge (around 7 judgements for a judge, 14 judgements per script) the marks are accurate to within +/- 1.3 for the reading essay (maximum of 20 marks) and +/- 2.8 for the writing essay (maximum of 40 marks).
Within 2 or 3 minutes per essay our judges are outperforming the marking metrics reported by Ofqual, of +/- 3 for a 20 mark essay and +/- 5 for a 40 mark essay.
How long did it take each teacher to judge a set of essays?
If it takes a teacher two or three minutes to judge an essay, how long will it take to judge a set of essays? Obviously that depends on how many pupils they teach! From our cohort it was clear there was a huge difference in pupil / teacher ratios judging in the schools, from 16 teachers judging the work of 10 pupils to 7 teachers judging the work of 152 pupils. The typical scenario, however, appears to be a teacher spending just over half an hour to judge one set of essays.
So, in summary, the teachers in our study judged a set of essays in just over half hour to a higher reliability than they can mark them .
If you would like to take part in judging your GCSE English essays next year, take a look at our national project.
