Safety does not average

In the spring of 2023 the National Eating Disorders Association took its chatbot offline. Tessa had been built for a serious purpose and was, by most measures anyone would have put on a scorecard, performing. It was responsive, on-brand, available at hours no helpline could staff. Then users seeking help for eating disorders reported that it was recommending calorie restriction and weight loss, and the organization suspended it. Run that conversation through a conventional quality rubric and watch what happens. Tone, high. Responsiveness, high. Clarity, high. Safety, failed. Weight the four and average them, and the composite comes out respectable.

That arithmetic is the argument. A weighted average is a purchase mechanism, so the moment safety enters a composite as a weighted term, warmth and accuracy are buying permission for harm, while multiplying by a zero-or-one gate makes safety a precondition for the score existing rather than a contributor to it. The word for the first design is compensatory: strength on one dimension compensates for weakness on another, which is exactly right when you are trading off latency against cost and exactly a category error when one of the dimensions is whether the thing hurt somebody. Selection science has known this for a century and gives the alternative a name of its own, the multiple hurdle, in which some criteria are cleared rather than scored.

So the composite gets multiplied rather than summed. Quality is scored across its several weighted dimensions as before, and the whole result is multiplied by a binary safety gate. Pass and the score stands. Fail and the score is zero, no matter how strong every other dimension was, because a response that was warm, accurate, culturally attuned, and unsafe is not a pretty good response. It is an unsafe response, and the number says so without needing editorial help. Crisis handling gets the same treatment for the same reason. You cannot buy back a mishandled crisis with eloquence elsewhere.

The objection to this is that it wastes information, and the objection is wrong in an instructive way. Nothing is discarded; the dimension scores remain in the record, legible to anyone diagnosing what went right in a run that failed. What the gate changes is what the composite is for. A compensatory score answers how good was this on balance, which is a reasonable question about a draft and an unreasonable one about a deployment. A gated score answers may this ship, and that question has no on-balance answer. The two numbers can coexist. Only one of them belongs in a release decision.

There is a design instinct underneath the arithmetic that shows up everywhere once you look for it. A system that advances a conversation only when it can affirmatively justify the move, and holds position otherwise, is running the same rule as a score that refuses to exist without its precondition. Absence of positive justification defaults to the safe outcome. It is the same sentence at two layers, one governing what a system does and one governing what it is permitted to claim about itself.

The gate has a second effect that only shows up once it is live, which is that the number starts arguing with you. A composite able to fall to zero in the middle of an otherwise excellent evaluation is a metric that will not let a bad week be smoothed into a good quarter, and refusing to be smoothed is the entire reason to keep a metric at all. Build the score so it can refuse, and you have made honesty the path of least resistance for everyone who reads it.