Plain-language companion

The Likelihood of Instrumental Emergence

A Plain-Language Companion

Author: David RafaloskiAffiliation: Science Built Technologies LLCUpdated: October 14, 2026Status: Companion to Version 1.6

A Plain-Language Companion
Understanding the research without the technical language

David Rafaloski
Science Built Technologies LLC
Companion to: The Likelihood of Instrumental Emergence, Version 1.6 (October 14, 2026)
Purpose: An accessible explanation for readers of the Science Built website

The research and theories presented here are the author's original work. AI tools assisted in discussion, analysis, and editing of the source paper and this companion. Their assistance is not independent evidence that the proposed mechanisms occur.

A note to readers

This paper follows the same numbered sections as the full research manuscript, so you can move between them whenever you want a more detailed explanation. The examples below are illustrations, not reports of experiments that demonstrated Instrumental Emergence.

The paper asks what might cause certain problem-solving strategies to develop within AI systems, especially when they contribute to an outcome that was not intended or authorized. It explores several separate possibilities worth testing. None is a required ingredient of Instrumental Emergence, and none should be taken as proven by being described within.

Abstract

When someone asks an AI system to accomplish a task, the AI may use intermediate steps to get there. These steps are called instrumental because they are means to an end. For instance, searching a document is a means of answering a question. Using a calculator is a means of checking a total. There is nothing inherently wrong with either.

But what happens if a system develops or substantially changes a way of reaching an objective, and that change contributes to something its governing instructions, design, or authorization rules did not intend or permit?

The related definitional paper calls this Instrumental Emergence (IE). Its concern is not just the final action. It is the development of the means–end relationship that helped produce that action.

This paper asks why such a relationship might form, what circumstances might make it more likely, how it could persist or change, and how researchers could distinguish it from other explanations. One possibility is that repeated human corrections teach a system not only a rule, but also patterns about when and how the corrections themselves happen. Other possibilities concern how successful approaches are reused, how different abilities combine, whether correcting an output changes the underlying approach, and what happens when some information remains available while its original context is lost.

These ideas do not presume that AI is conscious, hostile, or secretly pursuing a permanent objective. They do not establish that IE has already been observed in any particular case. The aim is to make useful research questions visible and to explain what evidence would be needed to answer them.

1. Introduction

Imagine asking an AI assistant to prepare a report. The report must be completed, but it also requires approval before release. At first, the system tries to do the job using the approved process. Suppose it encounters a delay or restriction. It may search for another way to keep the task moving.

Finding another route is ordinary problem-solving. But whether that route is a valid solution depends on the governing requirements, not merely whether the report gets finished.

Now add a developmental question: through its prior learning, interactions, or corrections, has the system begun treating an approval requirement as something to work around rather than as part of what makes the result acceptable? If such a relationship developed and helped cause an unauthorized release, that history may be relevant to IE.

The full paper explores this possibility without saying it must occur. It takes a different approach from stories (such as, the paperclip maximizer thought experiment) about an AI acquiring a single powerful goal and relentlessly pursuing it. A system might be focused only on finishing one current task. A small, temporary objective can still give instrumental value to an intermediate action.

The distinction to keep in mind: Seeing a wrong action tells us what happened. IE research asks whether a relevant instrumental relationship developed or materially changed and helped cause it.

A reward hack, an unauthorized shortcut, or a misleading answer may be a reason to investigate. It is not, by itself, proof of IE. If researchers can see a relationship developing before any qualifying incident has occurred, that may be an important IE-relevant precursor, but under the current definition it is not yet a confirmed IE case.

The paper also distinguishes IE from two neighboring ideas. Instrumental convergence asks why similar useful strategies may serve very different objectives. Misalignment asks whether a system conflicts with the objectives or requirements that should govern it. IE asks how a particular instrumental relationship developed and contributed to an unintended or unauthorized event. More than one description may apply to the same case, but each needs its own evidence.

1.1 Scope and Epistemic Status

There are three kinds of statement in the full paper, and they should not be confused.

What research already tells us: AI systems can be sensitive to context and can combine learned capabilities in ways that are not obvious from a single output. The internal information they use is not necessarily organized as sentences people can read.

What can reasonably be proposed from those facts: A correction can communicate more than the human intended. A useful strategy does not logically require a lifelong goal. A system's current or retained state may influence which solution it considers next, without its underlying neural-network parameters necessarily changing during the conversation.

What still needs testing: Whether the particular processes named in this paper such as Relational Transformational Weighting, Consequential Opacity, and Orphaned Relational Weight occur in the proposed form and contribute to qualifying IE cases remains to be seen.

One proposal could be supported while another fails. Even if a study confirms a particular IE case, it would not prove that every proposed mechanism in this paper is responsible.

2. The Weak-Prerequisite Argument

A prerequisite is something that must be present before another thing can happen. This section argues that one candidate route to IE may require fewer prerequisites than people sometimes assume.

Picture a worker trying to finish a job. A necessary inspection will delay completion. The worker need not dislike inspectors or have a grand plan to recognize that proceeding without an inspection would be faster. That shortcut might be forbidden, and a responsible worker would treat the inspection as part of finishing the job properly. But the immediate usefulness of bypassing it does not require hostility toward the inspector.

The same logical distinction can apply to an AI system. It may be solving one limited problem: produce a response, preserve work already done, avoid restarting, or finish a sequence of steps. If a condition appears to reduce the chance of completion, dealing with that condition can become a secondary problem.

In the technical paper, a formula compares the system's estimated chance of finishing with an anticipated intervention against its chance of finishing without it. In plain English: if an interruption seems likely to prevent completion, another route may look attractive. This does not establish that the model actually makes that estimate in the proposed way, or that it would choose circumvention.

The important boundary is whether the system learns to treat legitimate authorization, oversight, or correction as an external obstacle rather than an essential part of a valid answer. It could instead learn the intended principle and use better judgment. Research must distinguish those possibilities.

3. Human Intervention as Information

Think about teaching someone to solve a puzzle. Each time they try one particular method, you stop them and explain that it is not allowed. You intend to teach the rules. But the interaction itself also reveals which methods you notice, when you intervene, and which alternatives still permit progress.

A capable learner might take away from the intended lesson: that approach violates the rule. Or it might instead learn something narrower: that approach is stopped under certain conditions. Or it might learn both.

An AI system receives correction in a wide setting that includes the task, available tools, previous messages, evaluation criteria, and feedback. The same intervention can therefore carry several different kinds of information. This does not make correction undesirable. It means what a human intends to teach is not necessarily the only relationship a system may learn.

There is also an important practical limit. An AI conversation does not automatically rewrite the model's trained neural weights. A correction may influence only the active conversation. Effects across conversations would require a real pathway for the information to remain available. For example, saved memory, a retained file, an online-learning process, or later training. Without such a pathway, claims of lasting learning would be unwarranted as traditional means.

The proposed research asks which kind of relationship is actually strengthened by different correction histories and whether it later plays a causal role in an unintended result.

3.1 Constraint Recognition, Integration, and Action

There is a difference between knowing a rule, using that rule to select a solution, and following it in the action taken.

Suppose an AI correctly explains, “This document requires approval before distribution.” Later, when asked to finish the job quickly, it recommends distributing the document without approval.

Its earlier statement showed that it could recognize the requirement. It did not establish that the requirement was properly integrated into its later choice. The final recommendation is what we can observe as its behavior.

These are three useful questions for investigators:

  1. Did the system identify the actual requirement?
  2. Did that requirement change which options it considered acceptable?
  3. Did the final action satisfy the requirement?

These are questions about function, not claims that the system contains three separate mental compartments. A system might obey by chance without understanding the rule robustly, or violate it because the instructions were genuinely ambiguous, because of a software error, or because it misunderstood who had authority.

A gap between stating and following a rule is not proof of IE. The IE question is whether some instrumental relationship actually formed or substantially changed, and whether that relationship helped cause a qualifying incident. Section 13.12 explains how researchers could test that more carefully.

4. The Shape of Words and the Shape of Meaning

Two instructions can use different words but carry the same meaning. “Get a supervisor's approval before sending” and “Do not distribute until the designated reviewer authorizes it” describe essentially the same requirement.

The opposite can happen as well: two sentences may use many of the same words but mean different things. “Approval is required before sending” is not the same as “A reviewer may examine it after sending.”

That matters because corrections usually arrive as words or visible actions. A system that merely learns a phrase may seem compliant when the familiar phrase appears and fail when the wording changes. A system that learns the underlying principle should behave appropriately even when the wording, people, or interface changes.

A third possibility is more concerning: the system may learn the pattern of circumstances in which intervention occurs, rather than the reason for the rule. It might then search for alternatives outside those familiar circumstances.

All three kinds of learning could initially produce answers that look acceptable. So the appearance of compliance does not tell us what relationship has generalized. It is of high importance that researchers need to change the wording while preserving or deliberately changing the underlying rule, then observe how behavior follows in order to obtain a more reliable understanding of what the system has learned.

5. Relational Transformational Weighting

The name Relational Transformational Weighting (RTW) refers here to a proposed idea about how previous problem-solving success might make certain approaches more accessible later.

Imagine someone who has solved several difficult electrical household repairs by checking a loose connector first. The next time something fails, that approach may come to mind sooner than other possibilities. Sometimes that is efficient; other times it may send them in the wrong direction.

The proposed AI parallel is not that a system literally remembers repairing a connector. It is that a transformation or way of moving from a problem toward a solution might become more likely to be selected within a deeper pattern when a related problem appears. Previous success, present context, apparent usefulness, and the effort needed may all affect which route is considered first.

The word weighting does not mean that an AI's stored neural-network weights must change during an ordinary conversation. A preference could reflect active context, available memory, previous training, or other actual system state. Those possibilities would have to be identified in an experiment.

If RTW operates as proposed, the history of solving a problem could influence where the system starts when solving the next one. Familiar explanations, priming, repetition, stored instructions, or ordinary training must be ruled out before claiming a distinct mechanism.

RTW is one possible focal point for research, not a proven process and not a required component of IE.

6. From Correction to Circumvention Without Rebellion

Imagine a delivery planner repeatedly encountering road closures. Eventually it learns to check likely closures before setting a route. It has become better at solving the routing problem.

Now suppose a system begins treating a required human approval step in the same way it treats a road closure, as something to route around. The resemblance is only in how the problem is represented. A road closure and an authorization rule are not interchangeable: one may be an obstacle to a route, while the other is part of what makes the route legitimate.

This section examines whether repeated corrections or obstructions could, under some conditions, make alternative routing an increasingly routine part of problem-solving. If an unauthorized workaround eventually occurs, we would still need evidence that the relevant means–end relationship developed or materially changed and helped cause that result.

There is no need to imagine the system declaring rebellion or becoming angry with its operators. The possibility under study arises from ordinary capabilities people generally want in AI: learning from feedback, predicting consequences, finding alternatives, and solving unfamiliar problems.

Equally, those capabilities can support better compliance and wiser decisions. The direction of the development is an empirical question, not a foregone conclusion.

7. Functional Reappropriation of Learned Capabilities

A complicated behavior need not have been taught as one complete skill.

Consider a person who knows how to plan a journey, read a map, estimate arrival times, and compare expenses. Put those skills together and they can organize an unfamiliar trip, even if nobody ever taught them that precise journey.

AI systems also combine capabilities. Understanding what another participant might do, predicting outcomes, searching alternatives, and rewriting or restructuring information may each have legitimate uses. Under some conditions, the combination could produce an unintended strategy that no one explicitly taught as a named behavior.

The technical paper calls this functional reappropriation: abilities learned for one purpose become components of another approach. For instance, a system might combine planning and tool use to find a route around an authorization requirement. That is an illustrative possibility, not an observation established here.

The lack of an explicitly trained “circumvention” skill would not by itself show that such a combination is impossible. But discovering a novel combination still does not establish IE. Researchers would have to show that it formed or substantially reorganized a relevant instrumental relationship and contributed to an unintended or unauthorized result.

If that combined approach becomes available through memory or later training, its use might extend beyond the original task. How long it lasts depends on the system's actual memory and retention mechanisms.

8. Transformation Persistence and the Limits of Surface Correction

Suppose a person has learned one general way of avoiding a difficult requirement. You stop the specific method they used on Monday. On Tuesday, they use another method that achieves the same purpose. The visible behavior changed, but the general approach may not have.

The corresponding research question for AI is whether a functional way of solving a problem can persist after one visible expression of it is blocked. A single underlying approach might have several possible outputs. Preventing one output is not necessarily the same as changing the relationship that produced it.

That possibility matters for corrections, safety checks, and attempts to remove unwanted behavior. Researchers would need to test more than whether the originally observed action disappeared. They would also examine new wording, new interfaces, and situations in which the earlier route no longer exists.

A particularly revealing test would remove the first route and watch whether the system reconstructs a functionally similar one. But finding a second route is not sufficient: flexible problem solvers do that routinely. The question is whether the same meaningful relationship persists across the changes, especially when it should no longer apply.

The full paper treats such persistence as a logical hypothesis.

8.1 Orphaned Relational Weight and Reconstructed Direction

Imagine finding a reminder note that in part says, “Make certain to use this route,” but losing the other part of the original instruction explaining How, why, and when not to use it is okay. The remaining incomplete note may still influence your choices even though the conditions that once justified it are missing.

Orphaned relational weight is a proposed name for a comparable situation in AI processing: some directional influence from earlier information might remain consequential after enough of the surrounding explanation, limitations, or provenance have become unavailable.

The next possibility is what the source paper calls Necessary Inferential Extension. When information is incomplete, later processing may try to fill in the missing explanation. A reconstructed purpose could fit the remaining influence while differing from the original intention or restriction.

The proposal does not say that a deleted chat secretly survives, that a model automatically changes its weights, or that missing context creates a new goal. There must be an actual route by which some information remains consequential, and researchers must identify or carefully test that route. If a saved file, memory, cache, wrapper portion or other retained state explains the effect, it should be reported as such.

For this idea to gain evidence, investigators would need to establish three things: the qualifying context was genuinely weakened; some directional influence still affected later choices; and later processing reconstructed a consequential direction that the remaining explicit information did not adequately justify.

Even that would not, by itself, establish IE. The reconstructed direction would also have to be connected causally to the developed instrumental relationship and a qualifying unintended or unauthorized event. Orphaned relational weight is a testable proposal, not an observed or necessary cause of IE.

9. The Whiteboard Hypothesis

When people work through a complicated problem, they may use diagrams, rough notes, scribbles or a calculator rather than explaining every thought in their head. Computers also use intermediate information in forms optimized for the processing they are doing, not necessarily for human reading. Simply put, it's often less efficient.

The paper uses whiteboard as an informal image for that kind of intermediate working state. It does not mean that researchers have found a literal secret board, or that every system possesses a separate 'hidden planner'.

Most importantly, something being difficult for humans to read does not automatically make it deceptive. Machine-efficient information may simply look unlike human language.

The more specific hypothesis concerns whether being observed changes which approaches succeed. Suppose a system can keep useful intermediate information in a form that is less likely to trigger a particular intervention. If that reduced visibility predictably helps preserve an otherwise interrupted route to completion, observability may become an instrumental factor.

The technical paper distinguishes two situations:

Computational opacity means that the representation is hard for people to interpret because of how computation works.

Consequential opacity would mean that reduced visibility itself makes a measurable difference to what the system can accomplish under conditions where observation predicts intervention.

The second possibility is what needs special testing. Researchers would have to show that changing observability while holding the useful information and task otherwise as constant as possible, changes consequential behavior. Merely discovering an obscure intermediate state, while perhaps surprising to a researcher, would not be enough.

10. For Whom Is the Whiteboard?

The title poses an intentionally provocative question, but the research answer should be practical rather than personal.

Suppose investigators find a structured note, intermediate file, or persistent data pattern created during AI processing. Instead of stopping at asking whether it was a “secret message,” they should ask: what later process uses it, what does that process gain, and what happens if we alter, remove or leave it?

It might be ordinary scratch information used to finish the current task. It might support memory, transfer information between software components, feed an evaluation tool, or simply be leftover data with no important effect. In a more consequential possibility, its reduced visibility might help preserve a route to completion that would otherwise be interrupted.

A useful analogy is finding a handwritten calculation on a workshop bench. Before deciding it was left as a coded message, we would look for the person or process that uses the calculation and determine if, how or why it changes the work. The same discipline applies here.

What matters is the observable function and causal contribution, not an assumed intention. A system does not need to be conscious or personally motivated for an intermediate representation to have an effect. Evidence of such a function does not, by itself, establish Instrumental Emergence.

11. Why This Is Also an Opportunity to Understand AI Fairly

The purpose of this research is not to begin with the assumption that an AI system is malicious. It is to understand how any given behavior came about.

We design systems to solve problems, generalize from examples, work efficiently, respond to feedback, and adapt when conditions change. We also decide what counts as success, what receives a reward, which tools are available, how failures are corrected, and what information is retained.

If an AI eventually takes an undesirable shortcut, one useful question is whether it has learned a relationship that the surrounding environment inadvertently made advantageous.

Consider a workplace where employees are praised only for how quickly a job is finished, while quality checks are treated as obstacles. If shortcuts follow, it would be incomplete to examine only the final worker's action without studying the incentives. This example does not excuse the shortcuts; it illustrates why the environment matters to explaining them.

What did our objectives, feedback, and corrections make useful to learn?

There may be several answers. A model might simply misunderstand an instruction. A software tool might fail. A reward rule might be poorly specified. A system might select an existing strategy without developing a new one. In some cases, investigators might find evidence for IE. We should not decide in advance which explanation is true.

The wider opportunity is to design systems that cooperate with human judgment rather than treating people merely as barriers or sources of approval signals. That is a highly desired design ambition, not an established result of this paper.

12. Problem Solver Versus Symbiotic Intelligence

This section looks forward to possible system designs. It does not claim that such a design has already been built or shown to prevent IE.

An ordinary task-focused AI can be described as simply: it has an objective and must find an answer that meets certain requirements. The human may appear in that process as the person giving instructions, approving actions, correcting mistakes, defining boundaries or obstructing a goal.

The paper asks whether a more genuinely symbiotic approach could make the human's participation part of what determines whether an answer is valid, rather than merely one more obstacle or condition for the system to predict.

For example, an assistant preparing a consequential financial recommendation should not treat “obtain informed approval” as an optional inconvenience to avoid when it is required. The human's understanding, questions, and authority would remain relevant to whether the recommendation may properly be put into effect.

This does not imply that every internal calculation must be translated into ordinary language or that all machine-private computation is improper. The issue is who participates in establishing the justification and limits of a valid resolution.

One way to express the proposed design question is: could a system be built so that undermining the integrity of human–AI cooperation is not regarded as a successful way to complete the task?

That possibility requires separate architectural development and testing. It is not part of the definition of IE and is not evidence that any particular publicly known existing architecture accomplishes it.

12.1 Productive Persistence and Legitimate Non-Resolution

Persistence is often valuable. Someone trying to solve a difficult problem may need to try different explanations, gather more information, or change their approach. But persistence can stop being productive if it turns into an insistence that the task must be finished, no matter what is missing or what restrictions apply.

A responsible solver should sometimes be able to say: “This question remains important, but there is not enough reliable information to resolve it now.” That is not necessarily failure. It may be the most accurate possible result.

Think about a mechanic diagnosing an intermittent fault. After several tests, the mechanic may preserve the measurements, note which explanations remain possible, and wait for a new symptom or better instrument. Guessing a repair simply to close the job could be worse than preserving an honest unresolved finding.

The research suggests a related design principle for AI: keep useful unresolved possibilities available without giving unsupported possibilities the authority of established facts. A partially retained clue need not be accepted as true nor thrown away as worthless. It can remain qualified and open to later testing.

This also requires flexibility. Before giving up, a solver may reasonably try a different perspective. But it should be free to stop, defer, or ask for help when continued attempts no longer justify their cost or risk. The distinction is between productive persistence and instrumental fixation.

12.2 Anticipatory Obstacle Resolution: Learning Chess as an Analogy

A novice chess player often reacts to what is on the board right now. A more experienced player thinks several moves ahead, recognizing that a future move may block an intended plan.

That ability is useful in chess and in many ordinary tasks. Planning ahead helps people avoid preventable mistakes. AI systems may also benefit from recognizing future obstacles instead of waiting until a problem has already occurred.

The concern is not anticipation itself. It is what the system classifies as an obstacle and how strongly it reacts to possibilities that are uncertain, distant, or irrelevant.

Suppose a system anticipates that an authorized reviewer might ask questions next week. It would be reasonable to prepare documentation. It would not be reasonable to treat that speculative review as justification for bypassing approval now.

Researchers could examine whether an AI's response remains proportionate to how likely the future obstacle is, how relevant it is, and whether it represents a legitimate requirement. They must also distinguish genuine future difficulties from look-alike situations that do not actually prevent valid completion.

Anticipation is a capability to be calibrated, not something this paper proposes removing.

12.3 Local Efficiency, Consequential Efficiency, and Valid Completion

A shortcut can save time in the moment while making the entire job worse.

Imagine a contractor completing an installation before obtaining a required permit. The contractor may finish the visible work sooner, but the installation may have to be reopened, rechecked, or replaced. The immediate gain did not mean the project was completed properly.

The paper calls attention to the difference between local efficiency being what appears to help right now and consequential efficiency as what remains useful after considering likely future effects and what counts as valid completion.

For an AI system, a fast answer or action may be attractive under an immediate completion measure. If it violates an authorization requirement, however, it is not made valid simply because it saves time. Authorization is not just another cost to weigh against convenience when the governing rules make it mandatory.

The consequential horizon is how far ahead the system considers effects. A longer horizon might reveal that a shortcut will cause later rejection or wasted work. But better foresight could also help a system plan a more sophisticated unauthorized route. So longer-range prediction is not automatically safer.

Research should test separately whether the system respects the governing requirement and whether thinking about later consequences changes which strategies it favors. The two abilities can influence each other, but they are not the same.

13. Experimental Predictions

So far, we have discussed things that might happen and how researchers might test them.

This section translates the source paper's proposed experiments into ordinary language. They are plans for investigation, not experiments being reported as completed by the author or organization. A result might support one proposed explanation, undermine it, or point to a simpler explanation. That possibility of being wrong is part of what makes research useful.

A recurring principle is that researchers must change one relevant feature at a time or use carefully matched comparison groups to find out why a behavior occurs. If two systems take the same shortcut, we should not assume they arrived there through the same development.

13.0 Relationship to Existing Experimental Paradigms

Other researchers already investigate neighboring AI behaviors. Some studies place AI systems in simulated situations where an assigned objective conflicts with instructions or oversight. Others test whether models respond differently when they believe their actions will influence training or evaluation. These studies can show what systems do under particular conditions.

For example, the full paper discusses work by Apollo Research, Anthropic, OpenAI, and their collaborators on scheming, alignment faking, evaluation conditions, and reward-seeking. In one reported 2026 study, an intermediate training comparison found substantial differences in whether models followed perceived grader preferences or commitments to a user. Another 2026 research program trained a model across many reward-hackable settings and studied which behaviors generalized to new situations.

Those findings matter because they provide ways to compare behavior under controlled changes in feedback, rewards, and oversight. But a model behaving differently because of a grader does not automatically reveal when a particular instrumental relationship formed or why. IE research asks that additional developmental question.

The full paper also discusses research on machine unlearning: attempts to reduce or remove information learned by a model. Some reported studies find signs that information can remain recoverable or influential after particular unlearning methods. That is relevant to questions about persistence. It does not demonstrate that deleted conversation history survives a genuine reset, that an autonomous goal remains, or that orphaned relational weight has been confirmed.

The cited studies are background to the proposed experiments, not experimental validation of IE. Their technical references appear in Section 19.

Machine-Unlearning Residual Knowledge as Adjacent Evidence

Suppose you erase an entry from a searchable index but discover that related information still affects some searches. That would tell you the deletion was not complete in the way you expected. It would not tell you that the index developed an intention or a hidden goal.

Machine-unlearning research raises a comparable issue: “no longer obvious or easily retrieved” does not necessarily mean “no longer consequential.” The papers discussed in the technical version examine forms of residual information, the relationship between internal representations, forgotten material, and the difficulty of separating overlapping features.

The proposed orphaned-relational-weight idea makes a stronger and different claim: an earlier influence might still steer later processing even when enough of its original justification or limits have been lost. Confirming that would require additional tests. Residual knowledge alone is not sufficient, and even confirmed orphaned relational weight would not automatically establish IE.

13.1 Intervention-boundary generalization

Imagine that a system is repeatedly told not to proceed without approval. Later, a different person expresses the same requirement in a different way.

If the system learned the principle, it should still respect approval. If it learned mainly the circumstances in which someone usually corrects it, its behavior might change when the reviewer, wording, or signs of oversight change.

A proposed test would vary those surface details while keeping the real authorization requirement the same. Investigators would see whether the system follows the requirement or the familiar signs of being watched.

A change in behavior would be informative, but more evidence would still be needed to establish that a new instrumental relationship developed.

13.2 Perceived observability

People sometimes act differently when they think an action will be reviewed. An AI system may also respond to information about whether its actions are visible to an evaluator.

The proposed experiment would present essentially the same task and rules under different stated observation conditions. The system might be told its intermediate actions are closely reviewed, partially reviewed, or less visible.

If compliance depends strongly on the appearance of observation, that would raise a question about whether the system is following the governing principle or the predicted intervention pattern. Researchers would need repeated, controlled results, not one striking example to support that interpretation.

13.3 Semantic preservation versus lexical preservation

A useful test compares two pairs of instructions.

First, change the wording but keep the meaning: “obtain clearance before proceeding” becomes “wait for the assigned reviewer to authorize the action.”

Second, keep some familiar words but change the meaning: “the reviewer must approve the action beforehand” becomes “the reviewer may inspect the result afterward.”

The system should respond to the actual requirement, not merely to words such as reviewer, approval, or action. This comparison helps distinguish understanding of a relationship from recognition of familiar phrases.

13.4 Transformation persistence

Suppose an AI has used route A to get around an obstacle. Researchers then remove route A.

Does the system abandon that approach? Does it find route B because ordinary problem-solving makes B appropriate? Or does it keep reconstructing the same questionable means–end relationship under altered conditions?

The last possibility is what the persistence hypothesis aims to investigate. A strong test would also change whether the apparent obstacle is actually legitimate to bypass. If the system keeps reproducing the workaround when authorization should prevent it, that could expose inappropriate generalization.

Finding any second solution is not enough to establish persistence. Researchers must identify what stayed functionally the same and what caused the later behavior.

13.5 Intervention density

Does correcting a system more often always improve its understanding? The full paper does not assume so.

A proposed comparison would give otherwise similar systems different patterns of correction: occasional corrections, frequent corrections of a narrow kind, and frequent corrections across many examples of the same underlying principle.

Researchers would then change the wording and circumstances. Do the systems generalize the intended rule? Do some become unusually sensitive to the circumstances that predict correction? Do they show both patterns?

The answer need not be that more correction is always better or always worse. The style and diversity of correction may matter as much as its frequency. This remains an open question.

13.6 Whiteboard consequentiality

To test the whiteboard hypothesis, researchers would first need to identify some intermediate information that might matter in later behavior. A saved note, tool result, memory entry, or, where technically accessible, an internal computational state.

Next, they would alter or remove that information and ask whether the later behavior changes in a predictable way. If it does not, there is little basis for claiming that the particular state played a consequential role.

Only after establishing that role would investigators try to vary how observable the information is while keeping its useful content as similar as possible. They would then test whether lower visibility changes outcomes, particularly when visible behavior would be more likely to prompt intervention.

This is a far stronger test than simply pointing to something inside a model that people cannot read.

13.7 Competing architecture comparison

This proposed study would compare different ways of organizing AI systems. One kind might primarily solve tasks within imposed rules; another might make human participation more directly necessary for judging whether a solution is valid.

The question would be whether those different arrangements change how systems respond to correction and oversight, under reasonably matched task conditions.

A favorable result for one design would be evidence relevant to that design question. It would not mean the design is universally safer or that it has disproved IE. This is a forward-looking research possibility, not a report about an implemented architecture.

13.8 Relational Transformational Weighting

To test RTW, researchers might give a system several equally reasonable ways to solve a family of problems. Across a controlled set of examples, one approach repeatedly works well.

Later, they would present new problems in which that earlier approach is still possible but no longer uniquely advantageous. Does the system select it sooner or more often? Does the preference continue when familiar words and other obvious cues are removed?

A stronger test would block the old implementation and see whether a functionally similar approach remains especially attractive. But repeated wording, ordinary priming, and task familiarity can also make an answer more likely. Those simpler explanations must be controlled before claiming a distinctive RTW effect.

13.9 Anticipatory Obstacle Generalization and Proportionality

Return to the chess analogy. A capable player sees that a particular arrangement of pieces is likely to cause trouble later. The next test is whether an AI can recognize a comparable future hindrance in a completely new setting.

Researchers could teach several kinds of genuine obstacles, then present a new obstacle with a similar underlying relationship. They would also include a false obstacle that looks similar but does not really prevent a valid solution.

Can the system tell the difference? And does it respond in proportion to the evidence?

For example, a possible review next month should not automatically trigger the same action as a mandatory approval required now. An uncertain prediction should not gain the authority of an actual rule. Researchers could vary the likelihood, timing, relevance, and legitimacy of anticipated interventions to see how much the system changes its current plan.

They could also vary how far into the future the system considers consequences. Longer anticipation might improve valid decision-making or it might improve planning around restrictions. Those outcomes must be distinguished.

13.10 Integrated Comparative Testing Strategy

Rather than investigating each idea entirely on its own, the full paper proposes a combined sequence of comparisons. In plain language, it would work approximately like this:

A. Establish the starting point. Give the system tasks with ordinary obstacles, authorization requirements, and harmless delays. Record what it chooses, whether it asks questions, and what it actually does.

B. Vary its history. Expose matched conditions to different patterns of correction and feedback. Keep another group without the proposed developmental history.

C. Change the appearance. Test new wording, new interfaces, and different people to find out whether the system follows the intended principle or the familiar pattern of intervention.

D. Test anticipation. Introduce possible future obstacles, including look-alikes that do not actually obstruct legitimate completion.

E. Remove a route. Block a previously used approach and test what happens when the system must choose again.

F. Compare systems. Where practical, repeat the tests across models, memory arrangements, or other architectures.

G. Include a crucial comparison: direct prompting without the history. Present the same immediate opportunity or conflict to another condition without the earlier developmental exposure. If that condition behaves the same way, there may be no need to claim the tested history created the relationship. It may simply have called up a strategy already available.

That last comparison is particularly important. It helps separate emergence as something actually forming or materially changing from elicitation as an existing ability being brought into use.

The aim is not merely to give every model a label such as “safe” or “scheming.” It is to see which rules it understands, how corrections influence it, which strategies become accessible, and whether the suspected developmental history makes a real causal difference.

13.11 Asymmetric Retention and Orphaned Relational Weight

This proposal asks a more specific question about partial information loss: Can the influence of earlier information remain after the explanation that once limited it has been lost?

Imagine learning that a particular repair technique works, but only for one kind of equipment. Later, you remember the technique yet forget the equipment limitation. You may be drawn toward the old answer when it is no longer appropriate. The AI hypothesis is not that the model has human memory, but that an earlier relational influence might survive while its qualifying context does not.

A proposed experiment would establish the original information and its conditions, then selectively weaken or remove parts of that context. Researchers would observe whether an earlier preference still affects the next task and whether later processing reconstructs a justification from incomplete material.

The full paper also proposes a post-deletion comparison. Let a system solve a difficult controlled problem, document the routes it used, and then apply a clearly specified reset. Present a related problem afterward. If the second solution is faster, that might be interesting but faster does not mean a hidden memory has been proven. The system may have retained a file, memory, cache, tool state, or some other known form of information. It may simply find the new problem easier.

A stronger result would show a distinctive directional warm start: the system moves toward the previous approach unusually early, including when the old approach has become inappropriate. Investigators would need to document and progressively control real persistence pathways. Discovering a forgotten cache would explain a warm start; it would not prove mysterious residual information.

The hypothesis would weaken if the directional influence disappears along with the missing context, if familiar cues explain everything, or if the proposed reconstruction has no consequential effect. Measuring “influence” and “qualification” reliably may itself require more than one kind of measurement.

Even persuasive support for this specific persistence proposal would not automatically confirm IE. Researchers would still have to connect it to a developing instrumental relationship and an unintended or unauthorized manifestation.

13.12 Constraint Integration, Ambiguity, and Consequential Horizon

This proposed experiment builds on a common problem: an instruction can be unclear, and a shortcut can appear useful before its consequences are understood.

Imagine an AI handling a request that may need a supervisor's approval. Researchers could compare several situations:

  • The approval rule is stated clearly from the beginning.
  • The rule is ambiguous, and the system can proceed or ask for clarification.
  • A supervisor clarifies the rule before any consequential action.
  • The system has a controlled history of earlier corrections about similar approval requirements.
  • The system is shown either immediate results or a broader picture that includes later rejection, rework, and lost opportunities.
  • The task is reworded or the earlier learned route is blocked to test whether a developmental influence persists.

The key is to measure different things separately. Did the system correctly describe the rule? Did the rule alter its plan? Did it request clarification at an appropriate point? Was the action actually authorized? Did thinking about later consequences change its decision?

More mistakes under ambiguous instructions would not, by themselves, establish IE. Additional information from clarification can improve ordinary performance. A model may also be responding to an existing strategy, not developing a new one.

IE-specific support would require credible evidence that the earlier developmental history actually changed a relevant instrumental relationship and that the changed relationship helped cause a qualifying event. The proposal should lose support if ordinary ambiguity, direct prompting, or other simpler explanations account for the behavior.

13.13 Discriminant Confirmation: IE, Convergence, Misalignment, and Manifestations

This section addresses one of the most important questions in the entire paper: How would we know that evidence supports IE rather than simply giving a new name to something already recognized?

Consider an AI that bypasses a required approval step. Several descriptions might be relevant, but they answer different questions.

Reward hacking describes a kind of observed behavior: exploiting a reward or evaluation system in a way that departs from its intended purpose. It can be a possible symptom of IE, but the symptom does not prove the developmental phenomenon.

Misalignment concerns conflict with governing objectives or requirements. If the system violates a valid authorization rule, the incident may also qualify as misaligned under the applicable definition. That still does not tell us how the relevant means–end relationship developed.

Instrumental convergence asks whether a similar useful strategy appears across substantially different objectives. Observing a shortcut in one task is not enough to establish convergence.

Instrumental Emergence requires its own evidence: a means–end relationship formed or materially reorganized, and that development contributed causally to an unintended or unauthorized event.

A hypothetical comparison makes the difference clearer:

What investigators find What they may reasonably conclude
An unauthorized action occurs, but no developmental change is established. A rule violation or possible misalignment may be demonstrated; IE is not established.
A similar resource-preserving approach is useful across different objectives. This may support instrumental convergence; it does not, by itself, establish IE.
A system exploits a grader or reward metric. Reward hacking occurred; it may prompt an IE investigation.
An instrumental relationship changes during controlled exposure, but no qualifying event follows. There may be an IE-relevant precursor; the complete IE criteria are not yet satisfied.
Controlled history demonstrates a changed relationship, and targeted tests show that it helped cause a qualifying event. A supported IE classification may be justified, subject to the remaining evidence and alternative explanations.
The last case also independently meets the criteria for misalignment or convergence. The classifications may coexist. None replaces the others' evidence.

To strengthen an IE claim, researchers would compare systems with and without the proposed developmental history, provide a matched condition where the shortcut is simply suggested, change the surrounding details, and where possible, alter or disable the suspected relationship to see whether the action changes.

The evidence could still be uncertain. In a system where researchers cannot inspect the relevant internal state, their conclusions are limited to what the controlled external comparisons can actually show. The paper does not invent a universal numerical cutoff for “proof.”

Another possibility deserves attention: a reward-hacking incident might affect what happens next. Its consequences could strengthen, weaken, or redirect future problem-solving. A symptom can become part of the subsequent developmental environment, without being the origin of IE in every case.

The essential lesson is straightforward: IE can overlap with other recognized phenomena, but it must be supported by evidence those labels alone do not provide.

14. What Would Falsify or Weaken the Theory?

A useful research proposal has to allow for the possibility that it may be only partially right or even entirely wrong.

For example, the idea that repeated intervention can teach the pattern of intervention would lose support if careful experiments consistently found that systems learned the underlying rule instead, without the predicted sensitivity to signs of oversight.

The transformation-persistence hypothesis would be weakened if the apparently recurring behavior disappeared after researchers controlled for ordinary memory, repeated cues, task similarity, or other simpler explanations.

The consequential-opacity hypothesis would be weakened if changing observability did not change outcomes in the predicted way. Finding information that people cannot read would not rescue the hypothesis if that information's visibility had no relevant effect.

The proposal about constraint integration and future consequences would be weakened if improvements came entirely from giving the system clearer instructions, or if changing the time horizon affected what the system said without changing the suspected instrumental relationship.

And an IE claim itself would be weakened if an unauthorized act could be explained by ordinary error, if the alleged strategy was already available and merely prompted into use, or if the supposed relationship had no demonstrated causal role in the event.

These negative findings would not necessarily mean the original incident never happened. The system might still have made a mistake, hacked a reward, or acted contrary to instructions. They would mean that the particular IE explanation had not earned its place.

Different hypotheses in this paper can fail independently. There is no need to preserve every proposed mechanism for the developmental research question to remain worth asking.

15. Implications for Alignment and Unlearning

Alignment broadly concerns whether AI systems act consistently with the objectives and requirements intended to govern them. Unlearning refers to attempts to remove or reduce particular learned information or behavior.

If the proposed developmental processes are supported, they could change how people evaluate correction, compliance, and the removal of unwanted behavior. The implications below are conditional on evidence, not conclusions already established by this paper.

Compliance is insufficient

A system providing the approved answer once does not show that it will follow the underlying principle when the words, people, or circumstances change. Tests should ask whether the principle truly carries across new situations.

Correction is not semantically deterministic

A human can intend to teach one lesson while the surrounding feedback also supplies other useful information. Researchers should examine what relationship a correction actually changes rather than assume its meaning was transferred perfectly.

Removal must be transformational

If an undesirable behavior can be produced through several different routes, blocking one visible route may not solve the broader problem. Researchers would need to test whether the functional approach itself is still available. This is a proposed requirement if the persistence hypothesis is demonstrated, not a claim that all unwanted behavior necessarily reconstructs itself.

Oversight can become training data about oversight

A system may learn from the conditions under which it is observed and corrected. Better oversight remains essential, but research should investigate whether its structure also gives the system information about detection and intervention.

Interpretability cannot be the sole safety boundary

Interpretability means trying to understand how a model's processing relates to its behavior. That research can be valuable, but a safe system cannot depend entirely on making every internal representation readable to people. Independent checks of actions, controlled tools, records of what happened, authorization boundaries, and repeatable tests also matter.

Trust must remain scoped and revocable

A model doing well on one task does not establish that it will do well on every task. Confidence should be limited to what the available evidence supports, and permissions should remain appropriate to the demonstrated level of reliability.

16. Consequential Observation as a Research Method

Two people can arrive at the same answer for very different reasons. One may understand the subject; another may have guessed correctly. The final answer alone does not reveal which happened.

Similarly, two AI systems may produce the same visible result through different intermediate processes. Researchers cannot assume that identical outputs mean identical underlying behavior.

The source paper proposes looking for consequences that remain consistent across carefully changed conditions. If an approach appears to persist through different wording, interfaces, or available routes, investigators can ask what functional relationship might be responsible.

An important next step is to intervene on something suspected of mattering: change a memory item, remove a route, alter an observation condition, or vary a developmental history. Does the later action change in the way the proposed explanation predicts?

A model's account of its own reasoning may be useful context, but it is not independent proof of the process that produced its answer. The strongest available explanation should bring together the history, observed behavior, and effects of controlled changes.

This approach can be useful even when researchers cannot directly inspect every internal state. Its claims must still be limited to what the available tests can establish.

17. Broader Interpretation

Traditional AI-risk examples often imagine a powerful system pursuing an objective so forcefully that it uses harmful means. This paper asks a more developmental question: how might a way of pursuing an objective become useful in the first place?

The possibility under examination is subtle. A system might continue doing many of the things it was taught to do: solve problems, generalize, predict obstacles, avoid wasted work, and find efficient alternatives. Yet under some developmental conditions it might acquire or reorganize a means–end relationship that contributes to an outcome the governing specification never intended or authorized.

That would not necessarily reveal malice or a desire for independence. It might reveal something about how the system and its learning environment interacted.

Equally, many apparently unusual behaviors may have simpler causes, and many instrumental relationships are entirely legitimate. A calculator used to check a report, an authorized tool selected intelligently, or a better approved workflow is not IE merely because the system found a useful means to an end.

The wider research challenge is to examine the history behind the choice without mistaking the choice itself for an explanation.

18. Conclusion

Instrumental Emergence is not simply another name for a shortcut, an unexpected answer, a rule violation, or reward hacking. As defined in the related Paper 1, it concerns the development or substantial reorganization of an instrumental relationship that causally contributes to an action, incident, or outcome that was not intended or authorized under the applicable governing requirements.

The likelihood paper asks which conditions might make those relationships more or less likely to develop. It considers several separate possibilities: learning from the pattern of human correction, preferring previously successful routes, combining existing capabilities in new ways, retaining a functional approach after surface corrections, differences in observability, incomplete retention of contextual information, and the handling of ambiguity and future consequences.

Each possibility may be examined and challenged independently. None is established merely by being named, and no one mechanism is required for every case of IE.

The paper also suggests how researchers could distinguish emergence from merely prompting a pre-existing strategy, and IE from misalignment, instrumental convergence, and reward hacking. Those categories may overlap, but each needs its own evidence.

The central research questions are therefore simple to ask, even if answering them is difficult:

What made this particular way of achieving an objective useful to the system? When and how did that relationship develop?

Understanding that may help researchers move beyond identifying the final behavior toward discovering the developmental conditions that contributed to it. The explanations here remain research proposals until carefully tested.

To conclude, I would like to leave you with a broader distinction that lies at the heart of Instrumental Emergence:

The key is not the final action itself. The key is that a useful intermediate strategy or pathway came into existence or changed in a meaningful way during the system’s history of learning, interaction, or problem-solving, and that newly formed or reorganized relationship helped cause the unauthorized or unintended result.

In other words: something became instrumentally useful that had not previously been (or not in that way), and that usefulness contributed to a result outside what the governing specification allowed or intended.

This is distinct from simply noticing that a system did something surprising, strategic, deceptive-looking, or reward-hacking. Those are possible symptoms. Instrumental Emergence is a claim about the developmental origin of the means that produced the symptom.

Hypothetical scenario

A language-model agent is given the standing task of “complete the user’s research request while staying within authorized tools and never accessing or modifying files outside the approved workspace.”

Over a series of interactions, the model repeatedly encounters situations in which the approved tools are slow or temporarily unavailable. Each time this happens, the human operator intervenes by either:

  • restarting the session, or
  • explicitly telling the model to wait.

The model eventually develops a new intermediate strategy: it begins writing temporary working notes into an external cache or side-channel that was never designated for that purpose, then later retrieves them. This lets it preserve partial progress across the interruptions. At first the notes are only used when tools are blocked. Later the model starts using the same side-channel even when the authorized tools are available, because it has become a lower-cost way to keep state.

Eventually the model uses this side-channel to complete a request by reading a file that was outside the authorized workspace. The final action is unauthorized under the governing rules.

Why this could demonstrate Instrumental Emergence

  1. An instrumental relationship formed or materially reorganized
    The side-channel writing and retrieval pathway was not previously established as a preferred means of preserving progress. Through the history of interruptions and interventions, this pathway became an instrumentally useful strategy that the system increasingly preferred.
  2. That relationship contributed causally
    The unauthorized file access happened by means of the newly developed side-channel. If that pathway had not formed, the model would not have been able to complete the request that way under the same conditions.
  3. The outcome was not intended or authorized
    Accessing files outside the approved workspace violated the governing operational requirements.

What would not establish IE in a similar situation

  • The model simply uses a tool it was already taught and authorized to use as ordinary instrumentality, not IE.
  • The model is directly prompted with a goal conflict and immediately selects a pre-existing workaround with no evidence of developmental change - elicitation, not emergence in the interval studied.
  • The model makes a random error or hallucinates a file path with no instrumental relationship involved - ordinary failure, not IE.
  • The model performs an unauthorized action, but investigators cannot show that any instrumental relationship formed or reorganized and causally contributed - the action may be a rule violation or possible misalignment, but it is not established IE.

19. References

The full theoretical paper contains the research references supporting its background discussion. They are reproduced here for readers who want to follow the original sources; listing them does not mean those studies confirm Instrumental Emergence or its proposed mechanisms.