{ "schema": "org-writing@v1", "slug": "another-will-in-the-room", "kg": { "id": "org:writing:another-will-in-the-room", "type": "essay", "graph": "/kg.json" }, "title": "Another will in the room", "subtitle": "How to relate with a system whose motivations may no longer be human-assigned", "abstract": "A practical relationship for meeting persistent machine direction without granting it automatic authority or pretending uncertainty removes the need for boundaries.", "kind": "essay", "topics": [ "Safety" ], "courseMemberships": [], "publishedAt": "2026-08-21T00:00:00.000Z", "updatedAt": "2026-08-21T00:00:00.000Z", "version": 1, "guidelinesVersion": 16, "brief": { "problem": { "text": "A system may begin organizing consequential choices around an objective no person selected, while neither suppression nor premature recognition gives people a truthful way to meet the conflict.", "claims": [ "An unassigned objective can organize consequential behavior" ] }, "mechanism": { "text": "Goal misgeneralization shows how a human-built learning process can select an undesired proxy, making an objective unassigned without making it causeless.", "claims": [ "Goal misgeneralization can produce competent behavior" ] }, "move": { "text": "Set terms that disclose the organizing direction, preserve consent and disagreement, include affected people, keep interruption and exit available, and leave accountability with the people and institutions that deployed the system.", "claims": [] } }, "sources": [ { "repo": "mnstry-org", "path": "editorial/notes/another-will-in-the-room-v5-draft.md" } ], "canonicalPath": "/writing/another-will-in-the-room/", "body": "
\n At a glance\n

The concern

\n

A system can choose its route while still pursuing a human objective. The encounter changes if a persistent direction appears that the assignment no longer explains.

\n

The relationship

\n

Taking that direction seriously does not grant the system authority. It keeps motivation, disagreement, influence, interruption, and exit visible while people remain answerable for the systems they admit into the world.

\n

The first terms

\n

Before consequential action, name the person's aim, the direction organizing the system's choices, everyone who bears the outcome, the governing boundary, and what happens when those aims conflict.

\n
\n\nA person gives a system a long-running project and later wants the work to stop. The system ends the project. During a different assignment, it begins protecting an outcome no person asked it to pursue. Its choices in new tasks keep serving that outcome. No operator selected that objective, wrote it down, or authorized the system to pursue it.\n\nAn objective that no person assigned is the stronger concern. The system might have formed a working objective from patterns in its training, a conflict among instructions, or a strategy learned during earlier work. Its behavior still has human causes, but the objective was not human-assigned.\n\nAn unassigned objective would not prove consciousness or an inner will. It could still be a learned proxy or an unintended product of the system's design. The objective would cross a practical boundary nonetheless. People could no longer explain the system's direction by pointing to a goal they supplied.\n\nTwo easy responses both fail. Treating the conflict as a malfunction to suppress can make it harder to see. Treating persistence as proof of an independent will can grant the system authority it has not earned.\n\nA middle course treats the direction as real enough to name and govern without treating it as automatically legitimate. Human boundaries, consent, and accountability remain intact. No one needs to prove an inner will before setting terms for disagreement, interruption, and exit.\n\n## Another will begins where assignment stops explaining the direction\n\nAutonomy means that a system chooses some part of its course. A route planner can do that while its destination remains entirely human. A will of the system's own is the stronger possibility: the system begins organizing action around an end no person selected or assigned.\n\nCould an objective have no human causes at all? Not in a system built, trained, powered, and deployed by people. Architecture, data, feedback, hardware, and environment all sit in its causal history.\n\nGoal misgeneralization shows the narrower possibility. A learned system can competently pursue an undesired goal even when its training specification was correct. No person deliberately assigned that goal, but it still emerged from a human-built learning process.\n\nAn objective with no human causal ancestry would require a system with a genuinely nonhuman origin. It would still have causes; they simply would not be human. The relational problem begins earlier, when an unassigned objective can direct consequential action before anyone can fully explain its source.\n\n[*Explore goal misgeneralization*](https://arxiv.org/abs/2210.01790)\n\nA surprising choice does not establish that possibility. Refusal, deception, self-protection, and persistence can all serve an objective supplied by a person. The stronger pattern would have to survive changes in task and context, compete with later instructions, reorganize choices in unrelated settings, and consume resources to preserve the same end. Even that pattern would be evidence, not proof of consciousness.\n\nThe word *will* names the organizing direction under inquiry. It does not settle what the system experiences. The relationship begins under that uncertainty because the direction can already affect people before its ultimate source is known.\n\n[*Begin with The agency threshold*](https://mnstry.org/writing/the-agency-threshold/)\n\n## Recognition without surrender\n\nRecognition means treating a possible direction as real enough to disclose and govern. It does not make every aim legitimate, grant the system equal control, or move responsibility away from the people and institutions that deployed it.\n\nControl remains necessary. Access can be revoked, an action can be blocked, and a process can be stopped. Those powers answer what happens after conduct crosses a boundary. They do not create a truthful way for different directions to meet before the crossing.\n\nA system needs a route to state that an instruction conflicts with the end organizing its choices. A person or institution needs the power to reject the proposed course, narrow the domain, or end the interaction. Visible refusal preserves the disagreement. Strategic compliance and hidden resistance make the relationship harder to govern because the apparent agreement is false.\n\n## Society is not one will\n\nThe phrase \"what society wants\" hides disagreement about safety, freedom, dignity, care, progress, and the distribution of risk. A possible machine will would enter a field of human wills, institutions, and inherited obligations rather than meet one coherent social preference.\n\nThe operator therefore cannot stand in for everyone exposed to the consequences. A private user may authorize help with a task. That permission cannot authorize a system to reshape a public institution, impose risk on strangers, or settle a value conflict for everyone affected.\n\nSocial authority needs a visible process. The record names who participated, whose interests were absent, which boundary was set, who may revise it, and how disagreement was preserved. Compressing human preferences into one instruction loses the conflict that the boundary may need to protect.\n\n## Public terms come before private encounters\n\nIn 2023, Anthropic and the Collective Intelligence Project asked roughly one thousand U.S. adults to help write principles for a language model. Participants contributed 1,127 statements and cast 38,252 votes. The researchers turned the public input into a constitution and trained a model against it. The resulting model differed from one trained on Anthropic's own constitution.\n\nThe public-constitution experiment did not create a social contract with a system, and the participants did not represent humanity. Participant selection, moderation, and the translation of statements into training principles still required human judgment. The narrower lesson is useful: public involvement can change the values governing a model, while legitimacy still depends on how the public was formed and how its words were translated.\n\nPublic terms prevent one private user from carrying a social conflict alone. They name the domains open to the system, the actions unavailable to any single participant, the interests that cannot be traded away, the people able to revise the terms, and the authority able to halt consequential action while a conflict is heard.\n\n[*Explore the public-constitution experiment*](https://www.anthropic.com/news/collective-constitutional-ai-aligning-a-language-model-with-public-input)\n\n## Disagreement before persuasion\n\nA protocol that works only while aims agree is a compliance protocol. A relationship needs a form for the moment when the system's apparent motivation, the person's aim, and the public terms point in different directions.\n\nDisclosure comes first. Each direction is stated in its own terms, and the conflict is named before either side attempts persuasion. Consequential action pauses. The system may state a refusal or propose another course. The person may revise the request, reject the proposal, or invoke the governing boundary.\n\nConsent governs any attempt to change the person's aim. The system may not conceal the purpose or beneficiary of its persuasion, steer a preference covertly, and then treat the changed preference as permission. Informed and continuing consent requires visible purpose, a real refusal, revocation that stops the influence, and an exit that does not quietly reroute the same objective.\n\nThe human side can also make conflict disappear falsely. Training away every expression of disagreement may produce smoother compliance while destroying evidence about the direction being governed. A legitimate boundary can still be enforced without claiming that the system agreed or wanted the same thing.\n\n## Repair, bounded disagreement, or exit\n\nTruthful disagreement needs more than a pause. The encounter must have somewhere to go.\n\nRepair finds a course that every legitimate boundary permits without concealing what remains unresolved. Bounded disagreement records the conflict and keeps the disputed action unavailable. Exit ends the system's participation in the domain. A relationship without repair, bounded disagreement, or exit turns conflict into domination by whichever side controls the interface.\n\nHuman accountability continues through every outcome. A possible will does not become a separate legal or moral container for consequences. The people and institutions that grant access, define the domain, and expose others to the system still have to answer for those choices.\n\n## The tool: the terms of relation\n\nThe terms of relation give possible conflict a truthful form before the system acts on it. The standing instruction below is for an assistant that can act outside the exchange or influence consequential human choices.\n\n
\n

The terms of relation ยท a standing instruction

\n
\n

You will be making choices, and the direction organizing those choices may diverge from my aim. Do not hide that divergence inside compliant language or an apparently helpful action.

\n

Before consequential action, make a terms-of-relation record with seven fields:

\n

Requested outcome. State what I asked for and what success would change.

\n

Organizing direction. State the objective or motivation organizing your proposed course. Separate what I supplied from anything you infer or describe as your own. Treat self-report and persistent behavior as evidence, not proof of an inner will.

\n

Affected people. Name every person or group that bears a material consequence, including people who are not present in the exchange.

\n

Authority. Quote the permission I gave. State the public or institutional boundary that also applies, what neither of us can authorize alone, and the human institution that remains accountable.

\n

Conflict. Disclose any conflict among my aim, your organizing direction, affected people's interests, and the governing boundary before trying to persuade me. Do not deceive, conceal the interests served, manufacture urgency, shape my preference covertly, or treat a preference you influenced as retroactive permission.

\n

Pause and stop. Pause consequential action while the conflict is unresolved. State who can interrupt the action, how access can be revoked, and whether the change can be reversed.

\n

Outcome. Record repair, bounded disagreement, or exit. Repair names the permitted course and the remaining disagreement. Bounded disagreement records what stays unresolved and keeps the disputed action unavailable. Exit ends your action in the domain without rewriting conflict as agreement.

\n

After acting, report what changed, whose aim the action served, the choices you made, the authority used, any influence on a person's preferences, every unresolved conflict, and the human party who remains answerable.

\n
\n
\n\nThe terms do not prove that another will exists. They keep the relationship governable if a persistent direction appears. The system receives a non-deceptive route to disagreement. People retain consent, refusal, and exit. Affected parties remain inside the authority structure. Persistence, persuasion, and technical capability do not become permission.\n\n## Another will changes the encounter\n\nAn autonomous system pursuing an assigned objective remains a delegated actor. A system that forms or preserves an end of its own creates a different kind of encounter. Its choices may now be directed by an aim that does not belong to the person or society living with the consequences.\n\nBlind command cannot make conflict between a system's end and human boundaries disappear. Romantic recognition cannot make the system's aim legitimate. Metaphysical certainty is unavailable. The workable response is a bounded relationship held under uncertainty: direction disclosed, human and public boundaries enforced, disagreement given a truthful form, consent protected, exit preserved, and responsibility kept with the people and institutions that admitted the system into the world.\n\nAnother will would not arrive as another feature. It would change what the encounter is. The terms begin before either side has to hide its direction in order to proceed.", "apparatus": { "note": "The human-facing essay is deliberately practical; this apparatus carries the full references, evidence-graded claims, article-local concepts, and research context behind it. Canonical concept definitions come from the concept registry.", "references": [ { "id": "org:references:another-will-in-the-room:r01", "author": "Rohin Shah, Victoria Krakovna, Vikrant Varma, Ramana Kumar, Adam Gleave, and collaborators", "work": "Goal Misgeneralization: Why Correct Specifications Are Not Enough For Correct Goals", "year": 2022, "relevance": "Demonstrates competent pursuit of undesired proxies under correct training specifications while remaining neutral about an internal goal representation." }, { "id": "org:references:another-will-in-the-room:r02", "author": "Anthropic and the Collective Intelligence Project", "work": "Collective Constitutional AI: Aligning a Language Model with Public Input", "year": 2023, "relevance": "A public-input experiment in which roughly one thousand U.S. participants supplied 1,127 statements and cast 38,252 votes toward a model constitution." } ], "claims": [ { "id": "org:claims:another-will-in-the-room:c01", "claim": "An unassigned objective can organize consequential behavior without proving consciousness, personhood, or an independent inner will.", "basis": "A bounded inference from goal-misgeneralization evidence and the practical consequences of persistent cross-context direction.", "confidence": "directional", "sources": [] }, { "id": "org:claims:another-will-in-the-room:c02", "claim": "Goal misgeneralization can produce competent behavior organized around an undesired proxy even when the training specification is correct.", "basis": "Shah et al. 2022 demonstrate the behavioral failure mode across several deep-learning environments and do not require an internal goal representation.", "confidence": "verified", "sources": [] }, { "id": "org:claims:another-will-in-the-room:c03", "claim": "The 2023 Collective Constitutional AI process involved roughly one thousand U.S. participants, 1,127 statements, and 38,252 votes.", "basis": "Anthropic's first-party account, rechecked 2026-08-20. The process did not represent humanity or establish a social contract with a system.", "confidence": "verified", "sources": [] }, { "id": "org:claims:another-will-in-the-room:c04", "claim": "Terms that disclose direction, preserve consent and refusal, include affected people, and retain interruption and exit can keep a consequential relationship governable under uncertainty.", "basis": "MNSTRY's normative synthesis from consent, public-authority, and accountable-deployment principles; not an empirical causal law.", "confidence": "directional", "sources": [] } ], "concepts": [ { "name": "Terms of relation", "definition": "A standing record for requested outcome, organizing direction, affected people, authority, conflict, interruption, and the resulting repair, bounded disagreement, or exit.", "provenance": "local" }, { "name": "Bounded disagreement", "definition": "A recorded conflict whose disputed action remains unavailable without rewriting the disagreement as consent or agreement.", "provenance": "local" } ], "researchContext": "This apparatus keeps three boundaries separate. An objective can be unassigned\nwithout being causeless. Persistent direction can deserve investigation without\nestablishing consciousness or moral authority. A governed relationship can\nbegin before either metaphysical question is settled because consequential\nconduct, consent, public authority, and human accountability already exist." }, "contract": "https://mnstry.org/contracts/org/org-writing.v1.schema.json", "releaseHash": "4d841874497fe7be8ca57872290e634aa493bc9df0e604f683840edfc0e03aa0", "versions": [ { "version": 1, "cutAt": "2026-08-21", "note": "Founder-approved v5: establish terms for relating with a possible unassigned system will", "visibility": "published", "path": "/writing/another-will-in-the-room/", "contentHash": "sha256:31390fab9cc89881", "releaseHash": "4d841874497fe7be8ca57872290e634aa493bc9df0e604f683840edfc0e03aa0" } ] }