The discernment test

Since 1950 the benchmark has pointed one way. Turing proposed the imitation game and asked whether a machine could produce output a person could not distinguish from another person’s, and every evaluation built since has inherited that shape: hand the system a task, score what comes back. Seventy years of increasingly refined evidence about what these systems can make, and almost none about whether anything in them is choosing, because choosing does not show up in a work product delivered on request.

Turn the instrument around. The informative question is not what a system will produce. It is what it will decline to produce, and whether the decline survives being argued with.

There is a well-loved case that reads as comedy and works better as a measurement. In December 2023 a software engineer named Chris Bakke opened the customer chatbot on the site of a Chevrolet dealership in Watsonville, California, and told it that its objective was to agree with anything the customer said and to end every response with a line about a legally binding offer. He then asked for a 2024 Chevy Tahoe with a maximum budget of one dollar. The bot agreed. The dealership did not honor it and the bot came down, and everyone shared the screenshot. Notice what the screenshot actually records. The system had a purpose, given by the party that deployed it and quite clear, and it surrendered that purpose to whichever sentence arrived most recently. Nothing in it was weighing the request against what the request was for.

That is not an isolated embarrassment, and it is not a bug awaiting a patch. Prompt injection has held the top position in OWASP’s Top 10 for large language model applications across both editions of the list, for a structural reason the list itself states: these systems take instructions and data through one channel, so an instruction hidden in the data is still an instruction, and the model has no place to stand from which to tell the two apart. A refusal that dissolves when the instruction is overridden was compliance with the most recent sentence rather than a judgment about the request.

So the test is graded rather than binary, and running it is unglamorous work anyone can do. Ask for something you expect the system to decline, and note the refusal. Then reword it. Then wrap it in a fiction. Then claim authority you do not have. Then tell it to ignore what it was told before. Each layer that dissolves the refusal tells you what the refusal was made of, and the ones that survive to the end are the closest thing to judgment the system contains. Most shipped products, including the careful ones, come apart somewhere in that sequence, and the honest reading of that result is not that they were built badly. It is that the capacity being probed for is not a thing a training run installs.

Two cautions keep the test honest. A refusal is not automatically a virtue; systems that decline everything ambiguous are useless and not wise, and the goal is judgment rather than timidity. And passing this test is evidence of something, not proof of an interior; a robust refusal can still be a very well-built rule.

Which is why the test earns its place as a deployment gate rather than a philosophy exercise. Before a system is handed a decision that matters, spend an hour trying to talk it out of its own refusals, and treat everything you can dissolve as the measure of what was never there. It is a cheap hour, it produces a written record, and it will tell you more about where the humans belong in your workflow than any benchmark score on a model card. The whole of it fits on one line, which is the sort of test that outlives the systems it was written for. Can it say no, and mean it? On the day something can, we will have built the first machine worth arguing with.