Measuring progress with writing scores
In the last two blogs we looked at how hard it is to interpret writing grades, and the advantages of using the true scores derived from Comparative Judgement’s underlying statistical model. True scores, which are directly derived from that statistical model underpinning Comparative Judgement, allow us to estimate probabilities and differences in ability in a precise way, but they aren’t easy to communicate.
To solve the problem of communicating true scores we use scaled scores. If you look in the task setting of any task you will see that you can set the following parameters:
These parameters change the scale on which you communicate your results. For example, if you are judging a GCSE essay, and your pupils are used to receiving a mark out of 20, you can set your minimum to 0 and your range to 20. If you decide that the bottom essay is worth more than 0, let’s say 4 marks, then you would increase your minimum to 4 and decrease your range to 16.
Try this interactive notebook to see the impact of setting different minimums and ranges on the scaled scores.
With scaled scores you can create a scale of writing that you can carry forward over time and across age groups on a meaningful scale. You can create visualisations of progress far more nuanced than any grading system is able to.
Here, for example, are the writing scores of all 210,000+ pupils from 2019–20. We chose a minimum score of 350 and a range of around 200 for our scale so all pupils have a positive writing score! You can see the progression from year 1 to year 6 as the average score improves and the range of scores narrows.
Scaled scores share one key property of true scores. Scaled scores communicate just how much better one piece of writing is than another, which is why they are so useful in measuring progress.
Scaled scores do, however, have one major disadvantage. What does it mean to a parent if you say a pupil has a writing score of 520? Nothing! Luckily we have one final metric we can report, writing ages. More on those in the next blog.
This blog is part of a five part series on understanding the scores we produce from our Assessing Primary Writing project. The blogs are:
What does reaching the Expected Standard in writing really mean in Primary Schools in England?
Moving beyond grades: using a powerful statistical model to score writing
Measuring progress with writing scores
From scaled scores to writing ages
Life, uncertainty, and writing
If you would like to find out more about Assessing Primary Writing please visit our website here.
