The tautology trap

Validating an instrument against an outcome is the right instinct, and it is one of the few places a measurement of an interior life can earn its keep. Build the scale, find something in the world it ought to predict, correlate the two, publish the coefficient. The instinct fails in one specific way, and the failure is old enough to carry a proper name and current enough to be shipping inside evaluation harnesses this quarter.

Hubert Brogden and Erwin Taylor set out the taxonomy in Educational and Psychological Measurement in 1950. A criterion, they argued, can be deficient by omitting parts of what it is meant to capture, it can carry scale-unit bias, and it can be contaminated, which is what happens when something extraneous to the intended construct enters the criterion score. The most damaging entrant is the predictor itself. Their prevention rule was correspondingly strict, that nobody assigning criterion ratings may have any knowledge of the test scores, because a supervisor who has seen an aptitude result and then rates performance yields a validity coefficient that is partly a measurement of the test’s influence on the supervisor.

Push that leak to its limit and you arrive at the tautology. When a scale’s items and the criterion it is validated against are drawn from the same construct, the correlation between them was fixed at the moment the items were written, so a validation that looks strong is the instrument meeting its own reflection. No biased rater is required. Nothing has to be contaminated in transit, because the two quantities being compared were never independent to begin with. A scale asking whether you feel calm, correlated against a separate measure of calm, will report a handsome coefficient and will have learned nothing whatever about the world it was supposed to be measuring.

The best documented case in the wild is not a psychological scale at all. Michael Bastedo and Nicholas Bowman modeled the U.S. News college rankings in the American Journal of Education in 2010 and found that a school’s published ranking significantly affected the peer assessment scores it received in subsequent years, independent of changes in institutional quality and even of prior reputation. Peer assessment is an input to the ranking. So part of each year’s ranking is a measurement of the previous year’s ranking, laundered through the impressions of administrators who read it, and the stability everyone cites as evidence of the instrument’s validity is partly evidence of a loop.

Our own field has built that loop and calls it evaluation. A harness that shows a rater a system’s output, asks whether the response was helpful, and reports the aggregate as evidence that the system is helpful has validated nothing. It asked a question and printed the answer back with a different label attached. The criterion and the item are one sentence apart. Add that the rater usually sees only the response, without the context in which unhelpfulness would become visible, and the circle tightens rather than loosening, because the wording being scored has already been optimized against the judgment being solicited.

The discipline is unglamorous and it works. Write the criterion before the items, and require the criterion to be something obtainable by a person who has never seen the instrument. Did the user finish the task without asking again. Did the ticket reopen. Was the practice still going a month later with nobody prompting it. Criteria of that kind can embarrass an instrument, which is precisely their worth, and a team that arranges for its measures to be embarrassable on purpose is the only kind whose good numbers ever meant anything at all.