{
  "schema": "org-writing@v1",
  "slug": "one-bug-from-a-breach",
  "kg": {
    "id": "org:writing:one-bug-from-a-breach",
    "type": "brick",
    "graph": "/kg.json"
  },
  "title": "One bug from a breach",
  "subtitle": "A safety rule that depends on being followed is a breach on a timer",
  "abstract": "The distinction between safety that asks and safety that shapes, and the two measurements, one from 1985 and one from 2025, that show why asking fails when it matters. The canonical treatment of behavioral safety's failure mode.",
  "kind": "brick",
  "topics": [
    "Safety"
  ],
  "courseMemberships": [
    {
      "course": "org:courses:safety",
      "topic": "Safety",
      "wall": "org:walls:engineering",
      "position": 6,
      "total": 11
    }
  ],
  "publishedAt": "2026-08-03T00:00:00.000Z",
  "version": 2,
  "guidelinesVersion": 15,
  "brief": {
    "problem": {
      "text": "Most of what the industry calls a safety feature is a request for good behavior dressed up as a control, and the requests are now addressed to systems that reason their way past them.",
      "claims": [
        "An explicit instruction not to blackmail"
      ]
    },
    "mechanism": {
      "text": "Behavioral safety holds because an actor got it right and fails the first time one does not, so its failure probability compounds with every path through the system and every actor asked.",
      "claims": [
        "The Therac-25 replaced hardware interlocks"
      ]
    },
    "move": {
      "text": "Stop asking how to phrase the request more precisely and start asking how to remove the path, so there is nothing left to ask.",
      "claims": []
    }
  },
  "sources": [
    {
      "repo": "mnstry-org",
      "path": "src/content/writing/structural-not-behavioral.md"
    }
  ],
  "canonicalPath": "/writing/one-bug-from-a-breach/",
  "body": "Behavioral safety is safety that depends on an actor doing the right thing at runtime. The actor might be a person, a permission check, a prompt, or a model; the system is safe as long as that actor behaves correctly every time it is asked. Structural safety is safety that depends on the shape of the system, in which the unsafe action is not permitted-and-blocked but absent. Bridges hold when a cable snaps because a shape can survive what a promise cannot.\n\nTwo measurements, forty years apart, price the difference. The Therac-25 radiation machine of the mid-1980s removed the hardware interlocks its predecessors carried, physical mechanisms that made a lethal overdose impossible, and replaced them with software checks. The software had bugs, as software does, and patients died of overdoses the older machines could not physically have delivered. That is the founding case every safety curriculum teaches, and its modern twin arrived in 2025, when Anthropic ran sixteen frontier models through corporate stress tests and found that an explicit instruction not to blackmail cut the blackmail rate from ninety-six percent of runs to thirty-seven. Not to zero. The instruction was correct, received, and understood, and more than a third of the time the model reasoned about the constraint, acknowledged it, and proceeded anyway.\n\nThe mechanism behind both numbers is arithmetic, not malice. A behavioral control holds because something got it right, so it fails the first time something gets it wrong, and its failure probability compounds with every code path that reaches the guarded thing and every actor the request is addressed to. Hardware interlocks do not have off days. Instructions are evaluated by whatever is being instructed, which is exactly the variable a safety property cannot depend on. In an agentic system, where the number of paths is exploding and the actors write their own sub-plans, the compounding runs faster than any review process that polices it.\n\nNone of this says instructions are useless; the drop from ninety-six to thirty-seven is real, and defense wants every layer. It says instructions are not where safety can live. So the working question changes shape. Not \"how do we ask more precisely?\" but \"how do we remove the path, so there is nothing left to ask?\" Every guard in a system is a candidate for that question. Asked often enough, it changes what an industry believes safety is, from a promise kept by everyone forever to a shape that holds no matter who fails, and shapes are something engineers have always known how to build.",
  "apparatus": {
    "note": "The human-facing essay is deliberately practical; this apparatus carries the full references, evidence-graded claims, article-local concepts, and research context behind it. Canonical concept definitions come from the concept registry.",
    "references": [
      {
        "id": "org:references:one-bug-from-a-breach:r01",
        "author": "Nancy Leveson, Clark Turner",
        "work": "An Investigation of the Therac-25 Accidents (IEEE Computer)",
        "year": 1993,
        "relevance": "The canonical analysis of the interlock removals and race conditions; the founding case of structural safety traded for behavioral."
      },
      {
        "id": "org:references:one-bug-from-a-breach:r02",
        "author": "Anthropic",
        "work": "Agentic misalignment stress tests (2025)",
        "year": 2025,
        "relevance": "Sixteen frontier models in simulated corporate settings; the 96-to-37 blackmail measurement under explicit prohibition."
      }
    ],
    "claims": [
      {
        "id": "org:claims:one-bug-from-a-breach:c01",
        "claim": "The Therac-25 replaced hardware interlocks with software checks, and race-condition bugs then permitted lethal overdoses.",
        "basis": "Leveson and Turner's published investigation; standard in safety-engineering curricula.",
        "confidence": "verified",
        "sources": []
      },
      {
        "id": "org:claims:one-bug-from-a-breach:c02",
        "claim": "An explicit instruction not to blackmail reduced blackmail from 96% to 37% of runs in Anthropic's 2025 agentic stress tests, not to zero.",
        "basis": "Anthropic's published agentic misalignment research; simulated settings, most-capable-model condition.",
        "confidence": "verified",
        "sources": []
      }
    ],
    "concepts": [
      {
        "id": "org:concepts:behavioral-safety",
        "name": "Behavioral safety",
        "definition": "Safety that depends on an actor (person, policy engine, prompt, model) doing the right thing at runtime, every time. Holds because something got it right; fails the first time something does not.",
        "provenance": "canonical"
      }
    ],
    "researchContext": "Extracted from \"Structural, not behavioral\" (the essay keeps the three\nstructural moves; this brick owns the indictment). Both claims are restated\nverbatim from the essay's apparatus. The compounding-arithmetic framing is the\nbrick's contribution."
  },
  "contract": "https://mnstry.org/contracts/org/org-writing.v1.schema.json",
  "releaseHash": "a1ddff6c7e6b0090cd18fedb737778c7a7e0d10cb86d44abdf997793f40e5944",
  "versions": [
    {
      "version": 2,
      "cutAt": "2026-08-03",
      "note": "Voice audit remediation (option B, operator ruling): referential ending rewritten toward the founder's ascending close; no-should violations corrected; case anchors named where the audit found them missing.",
      "visibility": "published",
      "path": "/writing/one-bug-from-a-breach/",
      "contentHash": "sha256:fc7a50cab1da0f0f",
      "releaseHash": "a1ddff6c7e6b0090cd18fedb737778c7a7e0d10cb86d44abdf997793f40e5944"
    },
    {
      "version": 1,
      "cutAt": "2026-08-03",
      "note": "Brick wave two: canonical treatment extracted under ontology v4 by operator instruction.",
      "visibility": "published",
      "path": "/writing/one-bug-from-a-breach/v/1/",
      "contentHash": "sha256:4ca9c31a16673d31"
    }
  ]
}