What accuracy measures

In 2010 Chaoming Song, Zehui Qu, Nicholas Blumm and Albert-László Barabási published a result in Science that still gets quoted as a triumph of modeling. Working from anonymized mobile phone traces, they measured the entropy of each person’s movement and found a 93 percent potential predictability, with remarkably little variation across individuals no matter how far they habitually traveled. The frequent traveler was not meaningfully harder to call than the homebody. Read as a statement about models, that number is impressive. Read as a statement about people, it is a measurement of how much of a life runs on rails, and the two readings are not in competition. They are one number pointed in opposite directions.

That ambiguity is the whole difficulty with accuracy as a figure of merit for anything predicting a person. Prediction succeeds where behavior is habitual and constrained and fails where a choice is value-laden and the situation is new, which means an accuracy score is a reading of how little room the person had, and any system acting on its own predictions is positioned to improve that score by reducing the room. Nothing malicious is required for this. A recommender that narrows what someone sees makes their next action easier to call, and the improvement lands in the same dashboard cell that would have registered a genuinely better model.

We can quote this against ourselves, and we will, because the sharpest statement of it we have found is in a document we are responsible for. Among the research files relocated to restricted custody this August, and described publicly in our note on the test that corpus was never given, one file states the principle exactly. A model’s predictive accuracy, it says, is inversely proportional to the agent’s freedom, most accurate for constrained and habitual behavior and least accurate for value-laden choices in novel situations. Three sections later the same document sets the system’s function as searching for action sequences that steer a person’s trajectory into a region the system has defined as good, and calls that output non-directive. The finding and the machine built to defeat it sit in one file, under one set of review flags, written by one pipeline. A separate thesis elsewhere in our estate reached the destination independently and stated it more bluntly, that predicting a person perfectly destroys them, then drew the operational line where it belongs by banning recommendations of the form others like you from any session context.

So the discipline begins with refusing to report accuracy bare. A figure without a statement of what was being predicted is uninterpretable in the direction that matters, because 93 percent on tomorrow morning’s commute and 93 percent on whether someone leaves a marriage are not the same claim about a model and are emphatically not the same claim about a person. Publish the number with the constraint level of the behavior it was scored on, and watch what the number does over time in a closed loop, because accuracy that climbs while the system is also steering has a second explanation nobody wants to write down.

The rule that falls out is structural rather than behavioral, which is the only kind that survives contact with a roadmap. A system that predicts and a system that steers can each be graded honestly. A system that does both cannot be graded on its predictions, because it is in a position to author the evidence. Keep the two apart, say what the accuracy was about, and the number goes back to describing a model rather than quietly describing a narrowing life, which is the only version of it a person could be shown without being diminished by it.