The competence illusion

A human first draft announces itself. It is uneven, it hesitates at the hard part, the naming goes slack in the third file, and a reader can feel where the author stopped understanding. Generated work does none of that. It arrives uniformly fluent, consistently named, plausibly structured, and confident throughout, and the confidence is flat across the parts the model had grounds for and the parts it did not. Every instinct a reviewer developed reading human drafts is calibrated to a signal that is no longer being transmitted.

The failure this produces has a specific shape. A model adopts a false premise early, that an endpoint returns a timestamp when it returns a string, that a library exposes a method it never had, that a column is unique when nothing enforces it. Everything downstream is then built to agree with the premise, because agreement is what generation optimizes for, and the result is a subsystem that is internally consistent with its own mistake. The linter has no opinion about premises. The type checker validates the shapes the code declares, not the ones the world supplies. The tests, frequently written by the same model in the same session from the same assumption, pass emphatically and prove only that the code agrees with itself. Every conventional quality signal confirms the error rather than catching it, which is a strictly worse position than the one those signals were built for, because a signal that stays silent is at least honest about knowing nothing.

Then the prototype runs, and running is the strongest illusion of all. The thing does what you asked, in front of you, on the demo path. Behind the demo path sit the plural case, the empty case, the concurrent case, the hostile case, and a security posture assembled from whatever was most common in the training data, which is not the same as whatever is correct for your data. Confidence in the artifact tracks how convincing the demo was, and how convincing the demo was tracks nothing that matters.

What survives contact with this is a small and slightly boring set of habits, all of which move the check upstream of the artifact. Verify the premises before the behaviour, which usually means checking the actual response, the actual signature, the actual constraint, rather than the code’s belief about them. Read the tests first, since a generated test that mocks away the logic or asserts that true is true will pass forever and cheerfully, and a valid test is worth more than the implementation it guards. Put the critique in a different session from the authoring, because a critic that shares a conversation with the author shares the author’s premises and will defend them with the same fluency. Ask that session to explain the code back rather than to approve it, and listen for the sentence where the explanation quietly asserts something about the world that you know to be untrue.

The underlying correction is a small one and it costs nothing to make. Working is not a verdict, it is a symptom, and it has always been a weaker one than we let ourselves believe. What we lost was the accident that used to protect us, the fact that writing a thing slowly by hand made a person understand it. What we get to do instead is understand it on purpose.