{
  "schema": "org-writing@v1",
  "slug": "the-competence-illusion",
  "kg": {
    "id": "org:writing:the-competence-illusion",
    "type": "brick",
    "graph": "/kg.json"
  },
  "title": "The competence illusion",
  "subtitle": "A prototype that runs is evidence of fluency, and fluency is not soundness",
  "abstract": "Why the ordinary quality signals go quiet exactly when a generated system is structurally wrong, and what to check instead. The canonical treatment of the competence illusion.",
  "kind": "brick",
  "topics": [
    "Craft"
  ],
  "courseMemberships": [
    {
      "course": "org:courses:craft",
      "topic": "Craft",
      "wall": "org:walls:engineering",
      "position": 3,
      "total": 5
    }
  ],
  "publishedAt": "2026-08-03T00:00:00.000Z",
  "version": 1,
  "guidelinesVersion": 15,
  "brief": {
    "problem": {
      "text": "Generated work arrives carrying every signal a decade of practice taught practitioners to trust, and those signals were all calibrated against human error, which looks nothing like this.",
      "claims": [
        "coherence rather than correctness"
      ]
    },
    "mechanism": {
      "text": "A false premise adopted early is built on consistently, so the artifact ends up internally coherent with its own error, and each conventional check confirms that coherence instead of catching the premise.",
      "claims": [
        "internally consistent and pass conventional checks"
      ]
    },
    "move": {
      "text": "Verify the premises rather than the behaviour, read the tests before the implementation, and ask a separate session to explain the work back rather than to approve it.",
      "claims": []
    }
  },
  "sources": [
    {
      "repo": "mnstry-research",
      "path": "topics/engineering/monorepo/ai-assisted-dev-best-practices.md"
    },
    {
      "repo": "mnstry-research",
      "path": "topics/engineering/monorepo/solo-spiral-methodology.md"
    }
  ],
  "canonicalPath": "/writing/the-competence-illusion/",
  "body": "A human first draft announces itself. It is uneven, it hesitates at the hard part, the naming goes slack in the third file, and a reader can feel where the author stopped understanding. Generated work does none of that. It arrives uniformly fluent, consistently named, plausibly structured, and confident throughout, and the confidence is flat across the parts the model had grounds for and the parts it did not. Every instinct a reviewer developed reading human drafts is calibrated to a signal that is no longer being transmitted.\n\nThe failure this produces has a specific shape. A model adopts a false premise early, that an endpoint returns a timestamp when it returns a string, that a library exposes a method it never had, that a column is unique when nothing enforces it. Everything downstream is then built to agree with the premise, because agreement is what generation optimizes for, and the result is a subsystem that is internally consistent with its own mistake. The linter has no opinion about premises. The type checker validates the shapes the code declares, not the ones the world supplies. The tests, frequently written by the same model in the same session from the same assumption, pass emphatically and prove only that the code agrees with itself. Every conventional quality signal confirms the error rather than catching it, which is a strictly worse position than the one those signals were built for, because a signal that stays silent is at least honest about knowing nothing.\n\nThen the prototype runs, and running is the strongest illusion of all. The thing does what you asked, in front of you, on the demo path. Behind the demo path sit the plural case, the empty case, the concurrent case, the hostile case, and a security posture assembled from whatever was most common in the training data, which is not the same as whatever is correct for your data. Confidence in the artifact tracks how convincing the demo was, and how convincing the demo was tracks nothing that matters.\n\nWhat survives contact with this is a small and slightly boring set of habits, all of which move the check upstream of the artifact. Verify the premises before the behaviour, which usually means checking the actual response, the actual signature, the actual constraint, rather than the code's belief about them. Read the tests first, since a generated test that mocks away the logic or asserts that true is true will pass forever and cheerfully, and a valid test is worth more than the implementation it guards. Put the critique in a different session from the authoring, because a critic that shares a conversation with the author shares the author's premises and will defend them with the same fluency. Ask that session to explain the code back rather than to approve it, and listen for the sentence where the explanation quietly asserts something about the world that you know to be untrue.\n\nThe underlying correction is a small one and it costs nothing to make. Working is not a verdict, it is a symptom, and it has always been a weaker one than we let ourselves believe. What we lost was the accident that used to protect us, the fact that writing a thing slowly by hand made a person understand it. What we get to do instead is understand it on purpose.",
  "apparatus": {
    "note": "The human-facing essay is deliberately practical; this apparatus carries the full references, evidence-graded claims, article-local concepts, and research context behind it. Canonical concept definitions come from the concept registry.",
    "references": [
      {
        "id": "org:references:the-competence-illusion:r01",
        "author": "MNSTRY engineering research record",
        "work": "AI-assisted development best practices (2026), which names the competence illusion as functional prototypes masking structural fragility",
        "year": 2026,
        "relevance": "The term's home in our own record and the source of the failure inventory the brick works from: hallucinated premises, security naivety, and local optimization that violates global patterns."
      },
      {
        "id": "org:references:the-competence-illusion:r02",
        "author": "MNSTRY engineering research record",
        "work": "AI-augmented solo spiral development methodology (2026), on hallucination cascades and cross-examination testing",
        "year": 2026,
        "relevance": "The cascade mechanism, a false premise built on until the artifact is internally consistent with its own error, and the separate-session cross-examination the brick recommends in its place."
      },
      {
        "id": "org:references:the-competence-illusion:r03",
        "author": "Barry Boehm",
        "work": "Software Engineering Economics",
        "year": 1981,
        "relevance": "Why premise errors are the expensive class. A false premise adopted at specification time is the earliest possible defect and therefore the costliest to discover late, which is exactly what the illusion arranges."
      }
    ],
    "claims": [
      {
        "id": "org:claims:the-competence-illusion:c01",
        "claim": "Generation optimizes for coherence rather than correctness, so a false premise adopted early propagates consistently through everything built on it.",
        "basis": "The training objective and the hallucination-cascade pattern documented in both of our engineering research records; a mechanism claim about how the failure is shaped, not a measured error rate.",
        "confidence": "directional",
        "sources": []
      },
      {
        "id": "org:claims:the-competence-illusion:c02",
        "claim": "Artifacts built on a false premise are internally consistent and pass conventional checks, including linters, type checks, and model-written tests.",
        "basis": "Practitioner reports in both research records; the mechanism is straightforward, since each of those checks validates internal agreement rather than correspondence with the world, but the frequency is not measured here.",
        "confidence": "directional",
        "sources": []
      },
      {
        "id": "org:claims:the-competence-illusion:c03",
        "claim": "Models generate tautological or logic-mocking tests often enough that generated tests must be read before the code they guard.",
        "basis": "Repeated practitioner observation in our research records; a practice rule with observational support rather than a measured rate.",
        "confidence": "directional",
        "sources": []
      }
    ],
    "concepts": [
      {
        "id": "org:concepts:competence-illusion",
        "name": "Competence illusion",
        "definition": "A functional artifact whose fluency, internal consistency, and passing checks are read as evidence of soundness. A false premise adopted early is built on consistently, so the work ends up coherent with its own error, and every conventional quality signal (it runs, the linter is quiet, the tests pass) confirms the error instead of catching it. Worse than a silent signal, which at least admits to knowing nothing.",
        "provenance": "canonical"
      },
      {
        "name": "Cross-examination",
        "definition": "Asking a session that did not author the work to explain it back rather than to approve it, so that the premises are stated aloud where they can be checked against the world.",
        "provenance": "local"
      }
    ],
    "researchContext": "Extracted from \"The craft\" (essay parent), which keeps the narrative version.\nThe parent essay argues the illusion in a section; this brick is its canonical\nhome and owns the diagnostic detail the essay compresses, in particular the\ndifference between a human draft that signals where its author stopped\nunderstanding and a generated draft whose confidence is flat across grounded\nand ungrounded passages.\n\nAttribution note. The phrase competence illusion comes from our own\nengineering research record rather than from an external source, and the\napparatus says so rather than implying a literature that does not exist behind\nit. The claims are graded directional throughout because the mechanism is well\nobserved and the rates are not measured, and we would rather carry the grade\nthan borrow authority for a practice rule that stands perfectly well on\nobservation.\n\nBoundary with the sibling bricks. The verification inversion owns why\nchecking became the job; this brick owns why checking is harder than it looks\nand what specifically to check. Attention dilution owns the other decay, the\none that runs on a clock rather than on a premise."
  },
  "contract": "https://mnstry.org/contracts/org/org-writing.v1.schema.json",
  "releaseHash": "f16c249d311002e301fb700970e7cb2118bee30ff5fd8cae0627349a15ec18a7",
  "versions": [
    {
      "version": 1,
      "cutAt": "2026-08-03",
      "note": "Initial publication, craft wave",
      "visibility": "published",
      "path": "/writing/the-competence-illusion/",
      "contentHash": "sha256:a694719f32c8883c",
      "releaseHash": "f16c249d311002e301fb700970e7cb2118bee30ff5fd8cae0627349a15ec18a7"
    }
  ]
}