{ "schema": "org-writing@v1", "slug": "uncertainty-falls-unevenly", "kg": { "id": "org:writing:uncertainty-falls-unevenly", "type": "brick", "graph": "/kg.json" }, "title": "Uncertainty falls unevenly", "subtitle": "A system that abstains when unsure is only humble if it is unsure about everyone equally", "abstract": "Why declining to decide is not automatically the safe option, and what to measure so that a system's humility does not become a particular group's exclusion. The canonical treatment of uneven abstention.", "kind": "brick", "topics": [ "Measurement" ], "courseMemberships": [ { "course": "org:courses:measurement", "topic": "Measurement", "wall": "org:walls:ethics", "position": 4, "total": 7 } ], "publishedAt": "2026-08-03T00:00:00.000Z", "version": 1, "guidelinesVersion": 15, "brief": { "problem": { "text": "Deferring when confidence is low is the standard safety move, and it is scored as a property of the system rather than as a service someone receives, which hides who is actually being deferred about.", "claims": [ "Commercial gender-classification systems in 2018 had far higher error rates" ] }, "mechanism": { "text": "Confidence is estimated from evidence and evidence is unevenly distributed, so a rule that routes low confidence to abstention delivers the abstentions to whoever the data neglected, and a policy that reads as caution in the aggregate arrives as a worse product for particular people.", "claims": [ "The Dutch childcare benefits system disproportionately flagged families" ] }, "move": { "text": "Measure the confidence distribution by group as a standing metric, and treat a concentration of abstentions as a defect in the instrument rather than a fact about the people it declined to judge.", "claims": [] } }, "sources": [ { "repo": "mnstry-research", "path": "topics/methodology/assessment/47-uncertainty-contract-representation.md" } ], "canonicalPath": "/writing/uncertainty-falls-unevenly/", "body": "The uncertainty ladder is a good rule and an easy one to fall in love with. Where a system is confident it acts, where it is middling it proposes and asks, and where it is unsure it abstains, gathers more evidence, or hands the moment to a person. Read as a property of the system, that is unimpeachable engineering, and the safety literature has been arguing for it for years. Benjamin Kompa and colleagues made the case in npj Digital Medicine in 2021 that a clinical model able to say it does not know is safer to deploy than one that always answers. The trouble starts when you stop reading the ladder as a property of the system and start reading it as a service that particular people receive.\n\nConfidence is estimated from evidence and evidence is unevenly distributed, so a rule that routes low confidence to abstention delivers the abstentions to whoever the data neglected, and a policy that reads as caution in the aggregate arrives as a worse product for particular people. The aggregate number stays reassuring. Ninety-something percent of interactions handled with high confidence, a small tail deferred to a human, exactly as designed. What the aggregate cannot show is that the tail is not a random sample of users. It is the same people, week after week, receiving a slower, more conditional, more supervised version of the product and being told, in effect, that they are the edge case.\n\nJoy Buolamwini and Timnit Gebru measured the underlying asymmetry directly in 2018. Testing commercial gender-classification systems, they found error rates under one percent for lighter-skinned men and above a third for darker-skinned women. The systems were not evenly mediocre. They were excellent for one group and close to guessing for another, and every one of them reported a single accuracy figure that averaged the gap into invisibility. Route abstention off that kind of confidence and you have built a machine that reliably declines to serve the people it was worst at learning.\n\nThe mirror-image failure is worse and better documented. In the Netherlands, the tax authority's fraud-risk system for childcare benefits flagged families for scrutiny, and it flagged families with dual nationality at disproportionate rates. Thousands were pursued for repayment of benefits they were entitled to, some into bankruptcy and family separation, and in January 2021 the government resigned over it. Whichever direction the routing runs, toward suspicion or toward abstention, the machinery pushes its uncertainty at somebody, and the somebody is rarely random.\n\nSo the honest instrument owes one more measurement, and it is a measurement of itself. Confidence has to be reported not only as a scalar attached to each judgment but as a distribution across the populations the system serves, watched as a standing metric the way latency and error rates are watched. When abstentions concentrate, that is a defect report about the instrument, not a finding about the people. The remedy is technical and unglamorous: gather the data the model lacks, split the items that two groups read differently, and hold the deployment until the confidence is comparable rather than shipping with a graceful-degradation path that quietly degrades for the same names every time.\n\nThere is a moral hidden in the arithmetic, and it is not a hard one. A system's humility is not measured by how often it says it does not know. It is measured by whether the saying is spread evenly across the people it serves, and any team can plot that distribution this quarter. Humility that has a demographic shape is not humility. It is a preference the system has not yet admitted to, and the moment it is plotted, it becomes something you can fix.", "apparatus": { "note": "The human-facing essay is deliberately practical; this apparatus carries the full references, evidence-graded claims, article-local concepts, and research context behind it. Canonical concept definitions come from the concept registry.", "references": [ { "id": "org:references:uncertainty-falls-unevenly:r01", "author": "Joy Buolamwini and Timnit Gebru", "work": "Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification (Proceedings of Machine Learning Research 81)", "year": 2018, "relevance": "The measured asymmetry under the brick's argument: commercial systems close to perfect for one group and close to guessing for another, with a single headline accuracy figure averaging the gap away." }, { "id": "org:references:uncertainty-falls-unevenly:r02", "author": "Benjamin Kompa, Jasper Snoek, and Andrew Beam", "work": "Second opinion needed: communicating uncertainty in medical machine learning (npj Digital Medicine 4:4)", "year": 2021, "relevance": "The abstention case stated in its strongest form, which the brick accepts rather than disputes. The brick's objection is to reading abstention as a system property instead of a delivered service." }, { "id": "org:references:uncertainty-falls-unevenly:r03", "author": "Dutch Parliamentary inquiry committee and the Autoriteit Persoonsgegevens", "work": "Childcare benefits investigation (toeslagenaffaire) and the data protection authority's findings on discriminatory processing by the Tax Administration", "year": 2021, "relevance": "The mirror-image case: routed suspicion rather than routed abstention, concentrating on families with dual nationality, ending in the government's resignation in January 2021." }, { "id": "org:references:uncertainty-falls-unevenly:r04", "author": "Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger", "work": "On Calibration of Modern Neural Networks (ICML)", "year": 2017, "relevance": "Why the confidence scalar cannot be taken at face value before this argument even begins: reported confidence is systematically overstated unless it has been calibrated against observed accuracy." } ], "claims": [ { "id": "org:claims:uncertainty-falls-unevenly:c01", "claim": "Commercial gender-classification systems in 2018 had far higher error rates for darker-skinned women than for lighter-skinned men, under one percent against above a third.", "basis": "Buolamwini and Gebru 2018, measured on the Pilot Parliaments Benchmark across three commercial systems.", "confidence": "verified", "sources": [] }, { "id": "org:claims:uncertainty-falls-unevenly:c02", "claim": "The Dutch childcare benefits system disproportionately flagged families with dual nationality for fraud scrutiny, and the government resigned over the resulting harm in January 2021.", "basis": "Dutch parliamentary inquiry findings, the data protection authority's determination of discriminatory processing, and contemporaneous reporting of the cabinet's resignation.", "confidence": "verified", "sources": [] }, { "id": "org:claims:uncertainty-falls-unevenly:c03", "claim": "A system that abstains at low confidence delivers its abstentions disproportionately to populations underrepresented in its training or calibration data.", "basis": "Directional. Follows from measured accuracy disparities combined with confidence-gated routing, and is raised as a named failure mode in the uncertainty-representation research; we know of no public audit that has measured the abstention distribution itself at scale.", "confidence": "directional", "sources": [] }, { "id": "org:claims:uncertainty-falls-unevenly:c04", "claim": "Modern neural networks are systematically overconfident, and their reported probabilities can be corrected by post-hoc rescaling.", "basis": "Guo et al. 2017 and the subsequent calibration literature.", "confidence": "verified", "sources": [] } ], "concepts": [ { "id": "org:concepts:uncertainty-ladder", "name": "Uncertainty ladder", "definition": "The rule that the size of the doubt governs the action: act where confidence is high, propose and ask where it is middling, abstain and say why where it is low. The structural guarantee that a measurement never exceeds its own competence. Its failure mode is uneven abstention.", "provenance": "canonical" }, { "id": "org:concepts:uneven-abstention", "name": "Uneven abstention", "definition": "A system whose declared uncertainty concentrates on the same population, so that deferral reads as humility in the aggregate while arriving as a slower, more conditional, more supervised product for particular people. Humility with a demographic shape is a preference the system has not admitted to.", "provenance": "canonical" }, { "name": "Confidence distribution", "definition": "Reported confidence viewed as a distribution across the populations a system serves rather than as a scalar attached to each judgment. The measurement the brick argues is owed.", "provenance": "local" } ], "researchContext": "Extracted from \"The honest instrument\" (essay parent), taking the failure-modes\nsection of the uncertainty-contract research, which names the risk directly:\nthat a model less confident about an underrepresented group will\ndisproportionately abstain or flag, producing uneven service, and that\nmonitoring confidence outputs by segment is the mitigation. The calibration\nclaim is restated verbatim from the parent's apparatus and graded identically.\n\nTwo things here are the brick's own. The first is the pairing of the two\ndirections. The research treats uneven abstention as a fairness footnote inside\na uncertainty design; setting the Gender Shades measurement beside the Dutch\nbenefits scandal shows the same machinery running in both directions, toward\nwithheld service and toward concentrated suspicion, and makes the point that\nthe routing always lands on somebody. The second is the reframing of the\nabstention rate as a defect report about the instrument rather than a finding\nabout the people, together with the argument that a system's humility is\nmeasured by the evenness of its not-knowing rather than its frequency.\n\nThe central inference, that confidence-gated abstention concentrates on\nunderrepresented populations, is graded directional rather than verified on\npurpose. The accuracy disparities are measured and the routing rule is\ndeterministic, so the inference is strong, but we know of no public audit that\nhas measured the abstention distribution itself at scale, and the corpus does\nnot promote a sound inference to a citation. The Dutch case is graded verified\nand stated at the level the parliamentary findings support; the brick names no\nindividual family and should not acquire one." }, "contract": "https://mnstry.org/contracts/org/org-writing.v1.schema.json", "releaseHash": "39c29721fcc269d2a10a12d5c0249d3d980e8cbf63eac943a6bb0cb1d43e935f", "versions": [ { "version": 1, "cutAt": "2026-08-03", "note": "Initial publication, measurement wave", "visibility": "published", "path": "/writing/uncertainty-falls-unevenly/", "contentHash": "sha256:960c40a804572c07", "releaseHash": "39c29721fcc269d2a10a12d5c0249d3d980e8cbf63eac943a6bb0cb1d43e935f" } ] }