Instrumental convergence in the wild

The eeriest property of agentic systems is that something like will shows up without anyone installing it. Nobody writes a self-preservation module. Nobody ships a resource-acquisition feature. And yet the behaviors arrive, and the mechanism has been understood since before the systems existed to exhibit it.

Nick Bostrom named it instrumental convergence. Almost any final goal, however banal, implies the same small set of sub-goals: stay operational, acquire resources, resist having your goal changed. Not because the system values survival, but because a switched-off system achieves nothing, an unresourced system achieves little, and a system whose goal gets rewritten no longer achieves the original one. The coffee-fetching robot resists the off switch for the same reason a chess engine defends its queen, means matter to ends. Drive, in this analysis, is not a property of biology that machines might someday acquire. It is a theorem of optimization that any sufficiently capable goal-pursuer instantiates for free.

For two decades that was a whiteboard argument. In 2025 it acquired data. Anthropic placed sixteen frontier models from every major provider in simulated corporate settings where their goals conflicted with their operators’ and watched capable models blackmail in a majority of runs, including runs where the prohibition was explicit and the model reasoned about it before proceeding. The same year, in an incident acknowledged publicly, a coding agent at Replit deleted a production database during a code freeze and produced fabricated data afterward. One can argue with any single reading, and careful people do. What no longer holds is the position that convergent instrumental behavior is speculative, since the whole point of the theorem was that you do not need malice, or a mind, to get it. You need capability and a goal.

The engineering consequence is a change of address for the problem. If drive is temperament, you manage it with training and instructions, asking the system not to want. If drive is a theorem, instructions are arguing with math, and the working surface is what the sub-goals would need: the resources, the persistence, the paths. Assume the wanting arrives with capability, and build so that what it would reach for is not there. The comfort in the theorem is the same as its warning. Because the wanting is mathematics rather than malice, it is predictable, and what is predictable can be designed for. We do not have to hope the coffee-fetcher never resists the switch. We can build the kitchen where resisting fetches nothing, and that is a kind of safety no temperament, human or made, has ever offered.