Scores without confidence

In August 2010 the Los Angeles Times published effectiveness ratings for thousands of individual public school teachers, by name, derived from a statistical model of how much their students’ test scores had risen. Measurement specialists objected immediately, and the objection was not that the model was worthless. It was that the ratings had been printed as points. Each teacher’s estimate carried an uncertainty wide enough that many of the people sorted into different categories were statistically indistinguishable from one another, and none of that width made it into the ranking a parent read over breakfast.

That is the shape of the failure, and it is a failure of construction rather than of humility. An interval is not commentary on a score but part of it, so a point estimate released without one is not a cautious claim made quietly, it is an incomplete claim made confidently, and everyone downstream treats the missing width as zero. The number gets copied into a spreadsheet, then into a decision, then into a sentence about a person, and at each hop the doubt that never travelled with it is assumed not to exist.

The reassurance most instruments offer instead is a single reliability coefficient, usually Cronbach’s alpha, treated as a certificate. Klaas Sijtsma’s 2009 paper in Psychometrika is the standard citation for why it is not one. Alpha is a lower bound rather than an estimate of reliability, it rises simply because you added more items, and a high value says nothing about whether those items measure one construct or five. It is possible to build a questionnaire with an impressive alpha, no coherent thing being measured, and a page of scores that look precise to three significant figures.

The honest version of the same classical machinery does the opposite. It converts reliability into a standard error of measurement, and from that into a band around every score. Not your resilience is 62, but your resilience is near 62, give or take about 8, and that range is what the instrument can stand behind. Read plainly, the band says something a bare number cannot: here is the size of my own ignorance about you. The professional standards for educational and psychological testing have required exactly this reporting for years, which is worth remembering the next time a product ships an integer as though the requirement were paperwork.

The discipline gets sharper once you ask what the band is for. It is not decoration and it is not liability management. It is the thing that determines whether a comparison is allowed at all. Two scores whose intervals overlap have not been shown to differ, which means a ranking built from point estimates can order people the underlying evidence cannot order. It is also the thing that determines whether change over time is real: a person’s score moving from 62 to 68 means nothing if the instrument’s error is 8, and the disciplined system says so rather than congratulating anyone. Our own research notes reached the sentence we have not improved on. Scores without confidence decay into superstition, and the decay is fast, because a number that has shed its uncertainty is indistinguishable from a fact.

None of this argues for silence. It argues for a small change in what counts as a complete output, one that any team can make on a Tuesday: a score arriving without its width is a bug report, not a result. Wire the interval into the record beside the value, and the whole downstream chain inherits an honesty nobody has to remember to add. The width was never the caveat on the finding. It was half of what you found.