Moving beyond grades: using a powerful statistical model to score writing
In the last blog we saw how difficult it is to derive any meaning from a grade label which covers a wide ability range. The grade Expected Standard in writing in English primary schools at Key Stage 2 covers over half the cohort, so it is of little use if you wish to communicate the ability it describes or to use it to measure progress.
Comparative Judgement has three key elements that allows us to build a more detailed picture of writing ability:
- It relies on relative judgement rather than absolute judgement
- It involves multiple views by multiple people on the same pieces of work
- It uses a powerful statistical model to estimate scores from the multiple views
The scores that are derived from the statistical model have some magical properties. If you download your feedback from a Comparative Judgement task you will see a column which we call the true score, which typically runs from around -5 to +5. Here are the true scores for our study on Star Wars characters.
The magical thing about these scores is that if you plug them into an equation, you can see the probability that one piece of writing will be chosen as better than another piece.
You can try plugging some scores and seeing the probabilities change in this notebook.
Of course, we like to do the maths for you, so in the results panel on our website you can see the probabilities displayed. We have a help guide on how to access this panel here.
While the true scores have magical properties, however, they aren’t so useful for communication. For a start they include negative numbers, and we aren’t generally comfortable with telling a pupil they have achieved a negative score. Theoretically, a pupil could even achieve an infinitely negative score, which might be hard to explain to a parent.
In the next blog we’ll look at how we transform the true scores into scaled scores.
This blog is part of a five part series on understanding the scores we produce from our Assessing Primary Writing project. The blogs are:
What does reaching the Expected Standard in writing really mean in Primary Schools in England?
Moving beyond grades: using a powerful statistical model to score writing
Measuring progress with writing scores
From scaled scores to writing ages
Life, uncertainty, and writing
If you would like to find out more about Assessing Primary Writing please visit our website here.
