Uncertainty falls unevenly

The uncertainty ladder is a good rule and an easy one to fall in love with. Where a system is confident it acts, where it is middling it proposes and asks, and where it is unsure it abstains, gathers more evidence, or hands the moment to a person. Read as a property of the system, that is unimpeachable engineering, and the safety literature has been arguing for it for years. Benjamin Kompa and colleagues made the case in npj Digital Medicine in 2021 that a clinical model able to say it does not know is safer to deploy than one that always answers. The trouble starts when you stop reading the ladder as a property of the system and start reading it as a service that particular people receive.

Confidence is estimated from evidence and evidence is unevenly distributed, so a rule that routes low confidence to abstention delivers the abstentions to whoever the data neglected, and a policy that reads as caution in the aggregate arrives as a worse product for particular people. The aggregate number stays reassuring. Ninety-something percent of interactions handled with high confidence, a small tail deferred to a human, exactly as designed. What the aggregate cannot show is that the tail is not a random sample of users. It is the same people, week after week, receiving a slower, more conditional, more supervised version of the product and being told, in effect, that they are the edge case.

Joy Buolamwini and Timnit Gebru measured the underlying asymmetry directly in 2018. Testing commercial gender-classification systems, they found error rates under one percent for lighter-skinned men and above a third for darker-skinned women. The systems were not evenly mediocre. They were excellent for one group and close to guessing for another, and every one of them reported a single accuracy figure that averaged the gap into invisibility. Route abstention off that kind of confidence and you have built a machine that reliably declines to serve the people it was worst at learning.

The mirror-image failure is worse and better documented. In the Netherlands, the tax authority’s fraud-risk system for childcare benefits flagged families for scrutiny, and it flagged families with dual nationality at disproportionate rates. Thousands were pursued for repayment of benefits they were entitled to, some into bankruptcy and family separation, and in January 2021 the government resigned over it. Whichever direction the routing runs, toward suspicion or toward abstention, the machinery pushes its uncertainty at somebody, and the somebody is rarely random.

So the honest instrument owes one more measurement, and it is a measurement of itself. Confidence has to be reported not only as a scalar attached to each judgment but as a distribution across the populations the system serves, watched as a standing metric the way latency and error rates are watched. When abstentions concentrate, that is a defect report about the instrument, not a finding about the people. The remedy is technical and unglamorous: gather the data the model lacks, split the items that two groups read differently, and hold the deployment until the confidence is comparable rather than shipping with a graceful-degradation path that quietly degrades for the same names every time.

There is a moral hidden in the arithmetic, and it is not a hard one. A system’s humility is not measured by how often it says it does not know. It is measured by whether the saying is spread evenly across the people it serves, and any team can plot that distribution this quarter. Humility that has a demographic shape is not humility. It is a preference the system has not yet admitted to, and the moment it is plotted, it becomes something you can fix.