Worst measured at the edges

Classical test theory hands every person the same error bar, which is a convenient fiction and a consequential one. Item response theory drops the fiction. By modelling each question separately, how hard it is and how sharply it separates someone who has a trait from someone who does not, it can report how much the whole instrument actually knows at every point along the scale. That report is the test information function, and the first time you see one plotted, the shape is a small shock. It is a hill. Information peaks somewhere near the middle of the trait and falls away toward both ends, which means the standard error does the opposite: narrow where the crowd is, wide out where the people are unusual.

Precision depends on where the questions sit, so a scale assembled around the middle of a trait carries its tightest band where nothing is decided and its widest band at the extremes, which is where selection, screening, and alarm all happen. Nobody convenes a review over an average result. The decisions get made about the person at the top of the distribution and the person at the bottom, and those are exactly the two people the questionnaire was least equipped to describe. An instrument can be entirely sound and still be reporting, in effect, that this candidate is somewhere in the top fifth, give or take a fifth.

Consider what that does to a threshold. A program that admits the top five percent needs the instrument to separate the top five percent from the top fifteen, and at that end of the scale it very often cannot, because there were only ever two or three items difficult enough to discriminate up there. The cutoff still runs. The names still sort. What has actually happened is that a coin flip has been laundered through a decimal point, and the people on both sides of the line have been told something about themselves that the evidence does not contain. The same arithmetic runs at the other end, where a screening tool built to describe ordinary distress is asked to identify the person in danger.

Two disciplines follow, and neither requires abandoning the instrument. The first is reporting the error conditionally rather than on average. A single reliability coefficient averages the hill flat and publishes the mean as though it applied to everyone; the professional testing standards ask for the error at the score in question, which is the number that actually governs whether this particular verdict is safe. The second is building toward the edges on purpose. Adaptive testing exists in large part for this reason: rather than asking everyone the same forty questions, most of which tell it nothing, the instrument selects the next question to be the most informative one it can ask given what it already believes, and stops when its uncertainty is low enough. Two people finish at different lengths, not because one is better but because the instrument reached honest confidence about one of them sooner. Graduate admissions testing moved to computerized adaptive form in the 1990s on exactly this logic.

What the hill really tells you is which people the instrument has not yet earned the right to describe, and that is a gift rather than an embarrassment. A blind spot you can plot is a blind spot you can fix, by writing items hard enough and easy enough to see the ends of the range, or by declining the decision until you have. The alternative is not a better measurement. It is the same ignorance without the map. Any instrument that reports where it stops seeing has told you where to build next, and that is the beginning of an instrument worthy of the people at the edges.