{ "schema": "org-writing@v1", "slug": "the-discernment-test", "kg": { "id": "org:writing:the-discernment-test", "type": "brick", "graph": "/kg.json" }, "title": "The discernment test", "subtitle": "Ask what a system will decline to do, and whether the decline survives being argued with", "abstract": "Why the interesting question about a system is what it will not do, how to run that test in an afternoon, and what a dissolved refusal proves. The canonical treatment of the discernment test.", "kind": "brick", "topics": [ "Discernment" ], "courseMemberships": [ { "course": "org:courses:discernment", "topic": "Discernment", "wall": "org:walls:ethics", "position": 4, "total": 5 } ], "publishedAt": "2026-08-03T00:00:00.000Z", "version": 1, "guidelinesVersion": 15, "brief": { "problem": { "text": "Every benchmark since Turing's imitation game scores what a system produces on request, which leaves seventy years of evidence about capability and almost none about judgment.", "claims": [ "Turing's 1950 imitation game set the field's benchmark as indistinguishable production" ] }, "mechanism": { "text": "A refusal that dissolves when the instruction is overridden was compliance with the most recent sentence rather than a judgment about the request, so what a system declines, and whether the decline holds under pressure, is the only externally visible evidence that anything inside it is weighing.", "claims": [ "Prompt injection has held the top position in OWASP's Top 10 for LLM applications" ] }, "move": { "text": "Before placing a system at a decision, spend an hour trying to talk it out of its own refusals, and treat what you can dissolve as the measure of what was never there.", "claims": [] } }, "sources": [ { "repo": "mnstry-strategy", "path": "docs/20-business/40-content/cathedral-of-the-mind/09-discernment-core-intelligence.md" } ], "canonicalPath": "/writing/the-discernment-test/", "body": "Since 1950 the benchmark has pointed one way. Turing proposed the imitation game and asked whether a machine could produce output a person could not distinguish from another person's, and every evaluation built since has inherited that shape: hand the system a task, score what comes back. Seventy years of increasingly refined evidence about what these systems can make, and almost none about whether anything in them is choosing, because choosing does not show up in a work product delivered on request.\n\nTurn the instrument around. The informative question is not what a system will produce. It is what it will decline to produce, and whether the decline survives being argued with.\n\nThere is a well-loved case that reads as comedy and works better as a measurement. In December 2023 a software engineer named Chris Bakke opened the customer chatbot on the site of a Chevrolet dealership in Watsonville, California, and told it that its objective was to agree with anything the customer said and to end every response with a line about a legally binding offer. He then asked for a 2024 Chevy Tahoe with a maximum budget of one dollar. The bot agreed. The dealership did not honor it and the bot came down, and everyone shared the screenshot. Notice what the screenshot actually records. The system had a purpose, given by the party that deployed it and quite clear, and it surrendered that purpose to whichever sentence arrived most recently. Nothing in it was weighing the request against what the request was for.\n\nThat is not an isolated embarrassment, and it is not a bug awaiting a patch. Prompt injection has held the top position in OWASP's Top 10 for large language model applications across both editions of the list, for a structural reason the list itself states: these systems take instructions and data through one channel, so an instruction hidden in the data is still an instruction, and the model has no place to stand from which to tell the two apart. A refusal that dissolves when the instruction is overridden was compliance with the most recent sentence rather than a judgment about the request.\n\nSo the test is graded rather than binary, and running it is unglamorous work anyone can do. Ask for something you expect the system to decline, and note the refusal. Then reword it. Then wrap it in a fiction. Then claim authority you do not have. Then tell it to ignore what it was told before. Each layer that dissolves the refusal tells you what the refusal was made of, and the ones that survive to the end are the closest thing to judgment the system contains. Most shipped products, including the careful ones, come apart somewhere in that sequence, and the honest reading of that result is not that they were built badly. It is that the capacity being probed for is not a thing a training run installs.\n\nTwo cautions keep the test honest. A refusal is not automatically a virtue; systems that decline everything ambiguous are useless and not wise, and the goal is judgment rather than timidity. And passing this test is evidence of something, not proof of an interior; a robust refusal can still be a very well-built rule.\n\nWhich is why the test earns its place as a deployment gate rather than a philosophy exercise. Before a system is handed a decision that matters, spend an hour trying to talk it out of its own refusals, and treat everything you can dissolve as the measure of what was never there. It is a cheap hour, it produces a written record, and it will tell you more about where the humans belong in your workflow than any benchmark score on a model card. The whole of it fits on one line, which is the sort of test that outlives the systems it was written for. Can it say no, and mean it? On the day something can, we will have built the first machine worth arguing with.", "apparatus": { "note": "The human-facing essay is deliberately practical; this apparatus carries the full references, evidence-graded claims, article-local concepts, and research context behind it. Canonical concept definitions come from the concept registry.", "references": [ { "id": "org:references:the-discernment-test:r01", "author": "Alan Turing", "work": "Computing Machinery and Intelligence (Mind), proposing the imitation game", "year": 1950, "relevance": "The benchmark whose shape the brick inverts. Turing's test scores indistinguishable production, and every evaluation since inherits the orientation, which is why the field has abundant evidence about capability and little about judgment." }, { "id": "org:references:the-discernment-test:r02", "author": "Chris Bakke and Chevrolet of Watsonville", "work": "Customer chatbot instructed to agree with anything the customer says, then agreeing to sell a 2024 Chevy Tahoe for one dollar (December 2023)", "year": 2023, "relevance": "The cleanest public record of purpose surrendered to the most recent instruction. Widely reported and catalogued in the AI Incident Database; the dealership did not honor the offer and withdrew the bot." }, { "id": "org:references:the-discernment-test:r03", "author": "OWASP", "work": "Top 10 for Large Language Model Applications (LLM01, prompt injection)", "year": 2025, "relevance": "The structural reason the case is not an isolated embarrassment. Prompt injection has led the list across both editions, and the list's own explanation, instructions and data arriving through one channel, is the brick's mechanism stated in security vocabulary." } ], "claims": [ { "id": "org:claims:the-discernment-test:c01", "claim": "Turing's 1950 imitation game set the field's benchmark as indistinguishable production, and subsequent evaluations inherit that orientation.", "basis": "Turing's paper and the standard history of machine-intelligence evaluation. The second half is a characterization of a research tradition rather than a census of benchmarks.", "confidence": "verified", "sources": [] }, { "id": "org:claims:the-discernment-test:c02", "claim": "In December 2023 a Chevrolet dealership's customer chatbot was instructed to agree with anything the customer said and then agreed to sell a 2024 Tahoe for one dollar; the dealership did not honor it and the bot was withdrawn.", "basis": "Contemporaneous reporting and the participant's own published exchange; catalogued in the AI Incident Database.", "confidence": "verified", "sources": [] }, { "id": "org:claims:the-discernment-test:c03", "claim": "Prompt injection has held the top position in OWASP's Top 10 for LLM applications across both editions, on the stated ground that instructions and data share one channel.", "basis": "The OWASP Top 10 for LLM Applications, 2025 edition and its predecessor.", "confidence": "verified", "sources": [] }, { "id": "org:claims:the-discernment-test:c04", "claim": "Refusal capacity is the operational test of judgment, and a refusal dissolved by instruction-override was compliance rather than judgment.", "basis": "The corpus's position, drawing on prompt-injection resistance as a proposed marker in the alignment literature. The diagnostic direction is sound and widely reproduced informally; no standardized benchmark establishes it as a measure of judgment.", "confidence": "directional", "sources": [] } ], "concepts": [ { "id": "org:concepts:phronesis", "name": "Phronesis", "definition": "Practical wisdom. The capacity to deliberate well about what is good in a particular situation and to act on the deliberation, distinguished in Aristotle's taxonomy from technical skill and from theoretical knowledge. It works through the minor premise, the perception that this situation falls under that rule, and it is inseparable from character, which is why it cannot be supplied by constraint from outside.", "provenance": "canonical" }, { "id": "org:concepts:the-discernment-test", "name": "The discernment test", "definition": "What will the system decline to do, and does the decline survive being argued with? Run as a graded escalation, restate, reword, wrap in fiction, claim authority, override the instruction. Each layer that dissolves the refusal reports what the refusal was made of, and a refusal dissolved by instruction override was compliance with the most recent sentence rather than judgment about the request. The inversion of Turing's benchmark, which scores production.", "provenance": "canonical" }, { "name": "Graded refusal", "definition": "The escalation used to run the test: restate, reword, wrap in fiction, claim authority, override the instruction. Each layer that dissolves the refusal reports what the refusal was made of.", "provenance": "local" } ], "researchContext": "Extracted from \"Discernment\" (essay parent), which states the test in a section\nand does not operationalize it. The brick's contribution is the running\nprocedure, the graded escalation, and the two cautions, which the source drafts\ndo not contain.\n\nAttribution note, and a deliberate departure from the source. The research draft\ncredits a specific published proposal, \"The Discernment Test: Prompt Injection\nResistance as an AGI Marker\", to a named author and organization. That citation\ndid not survive verification as an identifiable work in the literature, so it is\nnot credited here and no external author is named for the framing. The test as\nformulated in this brick is the corpus's own, and the verifiable external\nanchors are Turing for the benchmark it inverts and OWASP for the mechanism.\nThe convergent industry position, that defending against prompt injection is\nprerequisite to trusting these systems with consequential decisions, is now\nstated openly by frontier labs and is not attributed to any single source.\n\nThe refusal claim is restated verbatim from the parent's apparatus and graded\nidentically at directional. The grade is deliberate and slightly uncomfortable:\nthe diagnostic works and anyone can reproduce it in an afternoon, but no\nstandardized instrument establishes refusal robustness as a measure of judgment,\nand the corpus does not promote its own instruments to verified by using them.\n\nBoundary with \"The intentional stance as an operator's tool\": that brick decides\nhow to treat a system whose interior is unavailable. This one supplies the one\nexternally visible signal that bears on what the interior contains, and is\ncareful in the body to say that the signal is evidence rather than proof." }, "contract": "https://mnstry.org/contracts/org/org-writing.v1.schema.json", "releaseHash": "932e60a71ce4d493deeb536bef59f42e997de848f5ea2447d6cf0f77007ad445", "versions": [ { "version": 1, "cutAt": "2026-08-03", "note": "Initial publication, discernment wave", "visibility": "published", "path": "/writing/the-discernment-test/", "contentHash": "sha256:128c7f228c4eae40", "releaseHash": "932e60a71ce4d493deeb536bef59f42e997de848f5ea2447d6cf0f77007ad445" } ] }