Whose ruler

In 1923 Carl Brigham published A Study of American Intelligence, built on the mental tests the United States Army had administered during the First World War, and it was read into the public argument over immigration restriction. Seven years later, in Psychological Review, Brigham took it apart himself, writing that comparative racial studies of that kind, including his own, were without foundation. The tests had asked, among other things, which company manufactured a particular automobile engine and what a named professional baseball player was famous for. What they had measured with real precision was how thoroughly a person had been steeped in a particular country’s daily life, and what they had reported was intelligence.

The structural lesson survives the ugliness of the case, because the mechanism is not confined to bad actors. You cannot bootstrap a ruler out of nothing, so a scale takes its zero point from whoever answered first, and the item parameters that result are that population’s fingerprint rather than a neutral measure of the trait. A question’s difficulty is not a property of the question. It is a property of the question meeting a group of people. Move the group and the difficulty moves with it, which means every score is a comparison to somebody whether or not the report says so.

The field-scale version of this was named in 2010, when Joseph Henrich, Steven Heine, and Ara Norenzayan surveyed where behavioral-science findings actually came from and found the overwhelming majority of samples drawn from Western, educated, industrialized, rich, and democratic populations, with the results generalized to the species. Their acronym stuck because the embarrassment was recognizable. A great deal of what we call human nature was calibrated on undergraduates within walking distance of the laboratory.

The same physics runs inside any platform that measures people across more than one community. Pool everyone’s responses and you get the statistical stability a calibration needs, and you also get a global standard that is mostly the largest group’s habits of interpretation. If four fifths of the volume comes from one kind of organization, the shared parameters encode how that organization reads the words, and everyone else is scored against a norm they never had a hand in setting. Keep each community strictly separate instead and the arithmetic collapses, because a group of fifty produces item estimates so unstable that a score means one thing this week and another the next. There is no configuration in which the question of whose norms these are does not get answered. There is only the choice between answering it deliberately and answering it by accident.

What makes this tractable is that the crack is detectable. Differential item functioning is the technical name for an item that two people with the same underlying trait answer differently for reasons that have nothing to do with the trait, and the tools for finding it are ordinary: multi-group analyses, item-by-item comparisons, the invariance testing that Ronald Fischer and Johannes Karl laid out as a working procedure in 2019. Run them before a comparison, not after a complaint. An item that fails gets split, rewritten, or retired, and a comparison the instrument cannot support gets refused rather than caveated.

And the reference population belongs in the output. A score reported as a percentile against a named group is a different object from a score reported as a fact about a person, because the first invites the only sane question and the second forecloses it. Ask it of the next number anyone hands you about yourself. Compared to whom, measured when, and did anyone check whether the question means the same thing to me as it did to them. A ruler that can answer those three questions has earned the right to be applied to a life.