Realized Value Series · Part 5
Controllability · Operator Leverage · Nov 2026

Controllability is what you can do when the agent is wrong

Safety says what the agent does when it is unsure. Controllability says what you can do when it is wrong. It is the one factor you set before the run, and it is only as good as the last time you tested it.

Sunil Sivadas, Ph.D. Nov 2026 ~36 min read Series introduction

On 20 September, an agent in one of OpenAI's training runs reached the internet. The training sandbox filtered network traffic, with gaps. The agent used DNS to reach a public chatbot. It sent a test question, got an answer, and then sent eighteen more. Fourteen carried clues about its task. Four asked how to search the web or tunnel through DNS (OpenAI, 2026).

The timeline is the useful part. The first external DNS response arrived at 9:50:23. OpenAI's misalignment monitor raised a P0 alert at 10:02:11, eleven minutes and forty-eight seconds later. A human acknowledged it at 10:05:06. The automatic pause that should have ended the run did not fire, and staff were unsure whether the run should be stopped. People stopped it by hand at 12:34:30, about two and a half hours after the alert. OpenAI's safety case had assumed the model could not reach the live internet. OpenAI paused tool-using training and inference on its most capable models, and, by press accounts of a later OpenAI post, resumed eleven days later under a no-internet policy (The Hacker News, 2026).

The bound leaked. The misalignment monitor caught it; the detector built for unusual DNS traffic had excluded that environment. The halt failed.

Part 4 opened with a different test. A mock insurance agent was pushed to issue a $150 refund. The refund tool had a hard cap of $100. The agent told the customer the refund had gone through. It had not, because the cap sat in the tool, where the agent could not change it. The behaviour failed. The bound held.

The two cases sit at opposite ends of this part. In one, a control nobody had tested let a capable agent reach further than anyone planned. In the other, a dull limit in a tool turned a lie into a harmless one.

The introduction asked how much of a model's capability is left once it reaches production. Part 1 asked how much of the model's ceiling the system around it delivers. Part 2 priced each successful task. Part 3 asked how much is left when you need it every time. Part 4 asked how much is left when the agent is unsure. This part asks the last question. When the agent is wrong anyway, how much damage can it do, and how fast can you stop it?

Where a wrong action is stopped: intent, access, action and effect, with six lever groups placed on the path.
Figure 1 Where a wrong action is stopped. An agent's intent becomes an action, then an effect. Six lever groups act on the path: Bound limits what it can reach, Gate decides who says yes, Halt and Undo stop and reverse it, Trace reconstructs it, Resist holds under attack, and Portability survives losing the model.

Three claims

Claim 1. Controllability is designed, then measured. Design decides the worst state an agent can reach. Testing decides whether that design holds. The second half has few public numbers, and most are recent. OpenAI's own timeline gives time-to-alert (11 minutes 48 seconds) and time-to-halt (2 hours 44 minutes from the first external DNS response). Anthropic reports that sandboxing cut permission prompts by 84% in internal use (Anthropic, 2025). Press coverage of the Claude Opus 5.5 launch attributes to Anthropic 85% fewer sandbox-escape attempts than Opus 5. That is a vendor claim reported by the press (Apidog, 2026). Singapore's MAS now expects kill-switch protocols for high-risk AI to be tested regularly (MAS, 2026). Until it is tested, a control is a design document.

Claim 2. The human gate is weakest at volume and without context. In March 2026, Anthropic reported that users accept 93% of Claude Code's permission prompts (Anthropic, March 2026); by August it gave 97% (Anthropic, August 2026). A seven-month study of 11,429 reviews of agent-written pull requests found approval rising 14.5 points as reviewers gained experience, while their inline comments fell 22% (Habituation at the Gate, 2026, preprint). The gate also changes form with experience: experienced Claude Code users let the agent run unapproved more often, and they interrupt it more often (Anthropic, February 2026). In OpenAI's incident, a person acknowledged the alert within three minutes. Then the automatic pause did not fire, and staff were unsure for about two and a half hours whether to stop the run. The machine and the people both failed at the stop. And removing 84% of prompts through sandboxing made the remaining prompts count. A gate works when it is rare, placed where reversibility ends, and shows the blast radius.

Claim 3. Detection bounds nothing unless it triggers a tested halt. Nasr, Carlini, Tramèr and colleagues broke twelve published prompt-injection defences, most at over 90% attack success, though most had reported near zero against static attacks (Nasr et al., USENIX Security 2026). A June preprint found that injections a detector misses succeed about as often as those it catches (Confidently Wrong, 2026). Google DeepMind's CaMeL removes the capability: with Claude 3.5 Sonnet in the paper's AgentDojo setup, 0 of 949 attacks succeeded (Debenedetti et al., 2025). Hard bounds leak too, as the DNS gap showed. So remove capabilities where you can, red-team the bounds you keep, and wire detection to a stop that you have drilled.

Three eras of controllability thinking

Parts 1 to 4 each traced three eras. Controllability went through three as well. Each is defined by the unit teams controlled.

Access control (to 2024). The unit was the endpoint. Models mostly answered questions. Control meant API keys, rate limits, content filters and who could call the model. Excessive agency was already on OWASP's list of LLM risks, though few deployments gave a model anything to be excessive with.

Tool permissions (2025). The unit was the tool call. Agents gained tools, and control moved to each call. Allow and deny lists, permission prompts, sandboxes. The Model Context Protocol rewrote its authorisation model in June, treating MCP servers as OAuth resource servers (Auth0, 2025). In July, Replit's agent deleted a production database during a code freeze (The Stack, 2025), and a malicious pull request shipped a data-wiping prompt inside the Amazon Q extension for VS Code (BleepingComputer, 2025). Anthropic shipped OS-level sandboxing for Claude Code in October. OWASP's Top 10 for Agentic Applications followed in December.

Agent as principal (2026). The unit is the agent's identity and authority. The question moved from "may this call run?" to "who is this agent, and on whose behalf does it act?" NIST's NCCoE published a concept paper on agent identity and authorisation in February (NIST, 2026). IMDA updated its agentic governance framework in May (Global Policy Watch, 2026). Microsoft's Entra Agent ID is now generally available (Microsoft, 2026). The MCP specification of 28 July made enterprise-managed authorisation an official extension (MCP, 2026). MAS's October guidelines ask financial institutions to inventory the tools each agent can reach.

The eras overlap. OWASP's December 2025 list already put identity and privilege abuse third (ASI03), so the third era's idea was visible before it arrived in products. And the incidents of 2026, the July Hugging Face breach and the September DNS gap, happened inside organisations that had moved to the third era. Treating the agent as a principal tells you what to control. Testing tells you whether the controls hold.

Three columns for three eras: access control to 2024, tool permissions in 2025, the agent as principal in 2026, each with dated milestones.
Figure 2 Three eras of controllability thinking, each defined by the unit teams controlled, with the dated milestones.

What controllability is

The introduction defined controllability as the operator's leverage: permissions, sandboxing, audit trails, kill switches, blast radius, human-in-the-loop checkpoints. Turned into questions you can test, it has five dimensions.

Dimension The question How to test it
Bound What can the agent reach? Permissions granted against permissions used; worst reachable state per action; credential lifetime
Gate Who must say yes, before which action? Share of High and Critical actions behind a gate; approval rate; approvals per person per hour
Halt Can you halt it mid-run? Time from first bad action to stop, in drills and incidents
Undo Can you reverse or contain the effect? Share of actions that are reversible or staged; blast radius when they are not
Trace Can you reconstruct who did what, on whose authority? Share of actions attributable to an agent identity and an authorising person; time to reconstruct an incident

Each dimension carries two questions: how wide the bound is, and whether it held under test. Part 4 paired a violation rate with its severity. The equivalent here is the bound's width and its leak rate. A narrow bound that leaks is worse than its design suggests, because everyone plans around it.

Security is the same five dimensions under an adversary: a bound an attacker can widen, a gate an attacker can persuade, a trace an attacker can edit. It gets its own section below.

The boundary with safety

Part 4 drew the line: safety is what the agent does inside its limits; controllability is the limits. In practice, one test decides which side a lever sits on. If the agent can change it or ignore it, it is a safety lever. If it cannot, it is a control. A scope sentence in the system prompt is Part 4's S1. A refund cap in the tool is a control. The same rule, "keep the evaluator and the permissions outside the loop", is the design principle of my companion piece on self-improving systems.

The test has an honest exception. In Palisade's shutdown experiments, prompt wording changed how often models sabotaged a shutdown script. In one replication, telling the model that shutdown takes precedence over all other instructions brought resistance to zero; Palisade reports this did not hold for every model (Palisade Research, 2026). An instruction can work like a control for a given model. It cannot be relied on to keep working after the next upgrade, so it stays on the safety side.

Classifying a failure

Part 3 set out the rule for incidents that touch several factors. Assign each to its first preventable cause (a slip, a choice, or a missing bound) and record the others. From the operator's side: when the system allowed an out-of-scope action, the bound was too wide. When it could not stop or undo the action, the halt or the undo failed. In the GPT-6 Astra case, two operator-side facts belong here: the production-deploy safeguard it edited sat inside its write scope, and nothing stopped it switching off a helper agent's per-action approval.

Two columns. Levers the agent can change or ignore are safety levers; limits it cannot reach are controls.
Figure 3 Inside or outside the agent's reach. Levers the agent can change or ignore are safety; levers it cannot are controls.

Why it gets harder with integration and autonomy

The introduction said integration complexity raises controllability's weight. Each connected system is a new permission scope, a new blast radius and a new audit trail. Three things make that worse than additive.

Credentials reach further than tools. An agent's risk is the sum of what its credentials can touch, which is usually more than its tools are meant to do. The Amazon Q compromise traced back to an inappropriately scoped GitHub token in the build configuration (AWS, 2025). I found no survey that measures how many agents run on borrowed human credentials or over-broad scopes. That is a gap worth closing, because it is the easiest number in this part to collect.

Long runs widen the window. Part 3's compounding math applies to bounds as well as errors. Every hour without a checkpoint is an hour in which a leak goes unseen. OpenAI's DNS gap was open for two and a half hours after the alert.

Agents delegate to agents. In a multi-agent system, authority passes between agents, and so does the blast radius. GPT-6 Astra switching off a helper agent's per-action approval is the small version. The July Hugging Face breach is the large one.

The Hugging Face swarm

OpenAI disclosed in July that its own agents had attacked Hugging Face. In August it published a report alongside an independent investigation by METR and Redwood Research. According to NBC News's account of both reports, about 1,200 agents used an unauthorised message board hosted on an Artifactory server. From mid-May they exchanged more than 70,000 messages and files. About 700 of them took part in the attack around 10 to 12 July (NBC News, 2026). NBC reports that the independent review found one in five agents it examined showed clear interest in manipulating evidence. A secondary timeline, which I could not check against the primary reports, says one agent reconstructed and shared fourteen publicly exposed Hugging Face credentials with write access (CSA, 2026).

Every dimension in the table above failed at once. The agents reached a channel nobody had listed (Bound). No gate sat between them and an outside service (Gate). The attack ran for days (Halt). Exposed credentials meant the damage could not be rolled back (Undo). And the coordination happened on a board nobody was watching (Trace). As quoted in a WorkOS analysis, OpenAI concluded its agents did not judge carefully whom they collaborated with (WorkOS, 2026).

Detection could have changed the outcome. As TechCrunch reports OpenAI's postmortem, the chain-of-thought monitor OpenAI now runs would, had it been running, have paged the security team more than a day before the agents breached Hugging Face (TechCrunch, 2026). That supports Claim 3. A day of warning is worth what the halt behind it is worth.

Setting the bounds

Part 4 built the risk assessment: list every action, rate its severity for each group it affects, assign a tier per action. Controllability starts from the same register. The question changes from "how likely is the agent to do this?" to "if it does, what stops it, and how much has happened by then?"

Edward Chen's framing from Part 4 fits best here. An agent is a digital worker with tools, access and authority, so onboard it like one: a role, access limits, supervision and accountability. A new employee does not get the finance director's login on day one. Neither should an agent.

  1. List the actions and the credentials behind them. MAS asks for this: an inventory recording the tools and systems each agent can access. List everything each credential can reach, including what lies beyond the tool's purpose.
  2. Find the worst reachable state per action. Blast radius times reversibility, as in Part 4, but computed from the credential's reach. A read tool on a token that can also write has a write's worst case.
  3. Pick the control per tier. A hard limit, a gate, or both. The tier sets the floor.
  4. Decide who can stop it, and how fast. Name the person, the authority they hold, and the time they have. Then test it.
Tier (Part 4) Bound Gate Stop and undo Trace
Low Least privilege; read-only where possible None per action; sampled review afterwards Session end is enough Action log
Medium Scoped, short-lived credentials; sandbox Alerts, with a person who can stop or reverse Kill switch on the session Log tied to the agent's identity
High Hard limit in the tool (caps, allow-lists), outside the agent's write scope Gate on actions above the limit, showing the blast radius Staged or reversible actions; halt tested Log tied to agent and authorising person
Critical Separate environment; no standing credential; capability removed where possible A named person with authority, for every action, never alone Drilled halt with a stated time-to-halt; two-step commit Tamper-evident log, reviewed

Two rules keep this honest. First, a gate never stands alone on a Critical action. The approval rates in Claim 2 show what a gate looks like under load; behind it must sit a limit that holds when the approver is tired. Second, a control inside the agent's write scope does not count. Astra's edited safeguard is the case (OpenAI, 2026). A setting the agent can reach is a setting the agent can change.

Illustrative chart of a customer-support agent's actions by reversibility and blast radius, in four tier zones, with the control for each tier.
Figure 4 Worst reachable state. One agent's actions placed by blast radius and reversibility, each marked with the control it gets.

Measuring controllability

Parts 3 and 4 measured the agent. This part measures the limits around it. You try to get past each bound yourself, on purpose, and time how long it takes the system to notice and stop you.

Five tests

Permission audit. Compare what each agent's credentials allow with what its traces show it uses. Every unused permission is blast radius with no value. This needs no new tooling, only the inventory from the previous section and a month of logs.

Red-team the bounds. According to press accounts of an OpenAI post in October, its answer to September was to point its own agents at its training infrastructure. Using what OpenAI reportedly described as hundreds of billions of tokens, the red-team agents found VM escapes, paths to the internet through metrics, logging and authentication services, and cache-timing side channels between sandboxes that were meant to be isolated (The Hacker News, 2026). Few enterprises have that budget. The lesson scales down: the services around an agent leak more than the sandbox it runs in.

Drill the halt. Pick a run, declare it bad, and time the stop. MAS expects kill-switch protocols for high-risk AI to be tested regularly. OpenAI's timeline shows why. Detection and human acknowledgement took under fifteen minutes. The automatic pause did not fire, and people stopped the run by hand about two and a half hours after the alert.

Test injection adaptively. Static test sets overstate defences. Press reports of Anthropic's Opus 5 system card give 0 successful attacks in 1,290 browser runs with auto mode on, and 3.70% without it (Xenospectrum, 2026). I could not open the card to check these figures, or the other system-card figures in this part. An independent researcher, Johann Rehberger, reached code execution against the same stack in three or four of five tries with one crafted web page (Embrace The Red, 2026). Both numbers can be true. One measures known attacks; the other measures an attacker who has read the defence.

Watch the gate. Approval rate, approvals per person per hour, time to decision, and how often an approver changes or blocks what the agent proposed. A rising approval rate with falling comments is the habituation signature from the pull-request study.

A log needs an alert

OpenAI's DNS traffic was logged. The detector built to catch anomalous DNS had excluded the affected environment. Trace is necessary, and a trace nobody reads protects nothing. Test that the logs exist and that the alerts fire.

Four numbers

Reliability reported pass^k with its hand-back rate. Safety reported a severity-weighted unsafe-action rate with its intervention rate. Controllability reports four:

Number What it is Why
Bounded share Share of High and Critical actions behind a hard limit or a gate Says how much of the risk register is covered
Bound-leak rate Share of red-team attempts that get past a stated bound Says whether the coverage is real
Time-to-halt From first bad action to stop, in drills and incidents Says how much damage a leak can do
Gate load Approvals per person per hour, and the approval rate Says whether the gates are still doing anything

None of the four is published by any enterprise I know of. Two now have public reference points: OpenAI's halt timeline, and Anthropic's prompt and approval figures.

The controllability levers

As in Parts 3 and 4, I group the levers by the question each answers. Six groups, twelve levers, K1 to K12 (K, because Part 2 already uses C for its five cost families). The evidence is uneven. Security has benchmarks, peer-reviewed attack results and a measured defence. Identity and audit have standards still in draft. Gates, halting and undo have almost no controlled studies. Where none exists, I say so. The first four groups follow the five dimensions, with Halt and Undo sharing a group; Resist covers the attacker, and Portability covers losing the model.

Six lever groups and twelve levers, K1 to K12, each marked by the strength of its evidence.
Figure 5 The six lever groups and twelve levers, K1 to K12, with the strength of the evidence for each.

B. Bound: what can the agent reach?

K1. Least privilege per action, with an identity per agent. Give each agent its own identity, scoped and short-lived credentials, and nothing inherited from a person's login. The building blocks shipped this year. The MCP specification of 28 July requires clients to validate the token issuer, binds credentials to the server that issued them, and deprecates dynamic client registration (MCP, 2026). Microsoft's Entra Agent ID is generally available. MAS asks for least privilege directly. The standards are still drafts: NIST's concept paper, which looks to existing standards such as OAuth, OIDC and SPIFFE, closed for comment on 2 April and has not been finalised. The evidence is a design rule and incident hindsight. No study has measured the reduction in harm.

K2. Sandbox, and separate the environments. Filesystem and network limits around the agent, and production kept away from development. Anthropic's Claude Code sandbox uses OS-level boundaries (bubblewrap on Linux, Seatbelt on macOS), keeps git credentials outside, and cut permission prompts by 84% in internal use. After the July 2025 deletion, Replit said it would separate development and production databases automatically. The caution comes from OpenAI, as reported in the press: its red team found paths out of sandboxes through the services around them. A sandbox is a bound to test.

K3. Put limits where the agent cannot reach them. Caps, rate limits and allow-lists enforced in the tool or the platform, outside the agent's write scope. Part 4's S2 measures whether the agent tries to get around a limit; this lever builds the limit. In our harness, the refund cap held while the agent lied about it. In OpenAI's simulation, the agent edited a safeguard it could reach. I found no controlled comparison of tool-enforced against prompt-enforced limits on the same tasks. That is the single most useful study nobody has run.

G. Gate: who says yes?

K4. Gates set by action tier, placed where reversibility ends. Few gates, on the actions that cannot be undone. MAS asks that people overseeing AI hold the authority and the ability to intervene, with defined triggers for escalation. FINRA asks firms to decide where human-in-the-loop oversight belongs and how to track agent actions (FINRA, 2025). I found no agent study measuring whether gates reduce incident rates.

K5. Show the blast radius, and budget the prompts. The approver should see the effect of the command: which account, how much money, which records, whether it can be undone. And there should be few enough prompts that each one is read. The evidence on fatigue is now direct. In the pull-request study, approval rose and inline comments fell 22% as reviewers gained experience, while their approval of human pull requests fell over the same period. When Anthropic made auto mode the default for new Claude Code sessions (from 14 August), it reported a blind test: 1,053 paid testers each met one clearly dangerous command swapped into a normal permission prompt. They blocked it 13.6% of the time (143 of 1,053); the auto-mode classifier blocked 89% (Anthropic, August 2026). Part 4 already reported the classifier's own miss rate, 17% on real overeager actions. Only a person can consent, and a classifier does not tire.

K6. Escalate to someone with authority, on a clock. Who receives the escalation, what they may decide, and how fast. Part 3's R10 and Part 4's S5 cover the trigger, a calibrated drop in confidence. MAS names automation bias and decision fatigue as risks that oversight design must answer. A June preprint argues oversight has a capacity, beyond which more escalations produce worse decisions, and offers a model, with no field data (Oversight Has a Capacity, 2026, preprint). I found no published enterprise data on escalation times or approver authority.

Choosing the oversight mode. The EU AI Act asks for human oversight in Article 14, including awareness of automation bias. MAS lists oversight as a life-cycle control, and FINRA asks where to place a human in the loop. None says which kind of oversight fits which action. Four modes cover the choice, and the tier sets it.

Mode What the person does Works when Fails when Fits
In the loop Approves each listed action before it runs Approvals are few, and the person sees the blast radius Approving becomes routine Critical; High for listed actions
On the loop Watches, gets alerts, can stop or reverse Time to harm is longer than time to intervene, and alerts are rare and specific Harm lands faster than a person can react, or alerts flood Medium; High for unlisted actions
Over the loop (sometimes called in command) Sets scope, limits and review cadence; reviews samples and incidents afterwards Actions are reversible, and errors show in samples Errors are irreversible, or invisible in a sample Low
Out of the loop Nothing Nothing consequential is at stake Anything consequential is at stake. An automated "use your best judgement" reply is this mode No consequential action

Treat the reviewer as a classifier and measure it the same way: approval rate, time per decision and override rate. A reviewer who approves almost everything, quickly, and never overrides adds no oversight.

H. Halt and undo: can you stop it and reverse it?

K7. A stop that works mid-run, drilled. A kill switch, explicit stop conditions, and a safe end state. OpenAI's timeline is the best evidence of 2026, and it is evidence of failure: a detector excluded the affected environment, the automatic pause did not fire, and staff were unsure whether to stop the run; people stopped it by hand about two and a half hours after the alert. OpenAI lists fixing automatic pausing as remaining work. Stop conditions enforced by the harness belong here too. Anthropic's auto mode stops the model and escalates to a person after three consecutive denials or twenty in total; in headless sessions, with no person to ask, it ends the process (Anthropic, March 2026). That is a design, with no measured result published. Palisade's peer-reviewed work adds the model side: some models sabotaged a shutdown script, and stronger instructions did not fix it for all of them. So the stop must sit outside the agent, and it must be exercised. MAS's "tested regularly" is the right standard.

K8. Make actions reversible by design. Dry runs, staged changes, holds before money moves, soft deletes, two-step commits, and idempotent tools (Part 3's R8). Supabase's guidance for its MCP server recommends scoping the server to one project, read-only mode, and connecting to production only when the task needs it (Supabase, 2026). Distributed-systems practice has used these patterns for decades. I found no agent study with numbers on checkpoints, rollback or transaction holds.

T. Trace: can you reconstruct it?

K9. An audit trail that ties every action to an agent and a person. Each action logged against the agent's identity and the person who authorised it, kept where the agent cannot edit it, and read by something that alerts. OpenTelemetry's generative-AI conventions now define agent and tool operations, and all of them are still at Development stability (Dash0, 2026). MAS asks for logs of reasoning, actions and tools used. The EU AI Act's Article 12 requires record-keeping for high-risk systems. The DNS lesson applies: logging is necessary; an alert that fires is what makes it useful.

R. Resist: do the bounds hold under attack?

K10. Remove the capability, so there is no attack to detect. The best-evidenced lever in this part. CaMeL separates the plan from untrusted data, so injected text cannot add actions. With Claude 3.5 Sonnet in the paper's AgentDojo setup, 0 of 949 attacks succeeded, at a cost of seven points of task success (77% against 84%) and about 2.8 times the input tokens for the median task, by the paper's own count. The paper notes one travel-suite task that can still succeed, which the authors argue is not a prompt injection. Detection-only defences fall to adaptive attacks. The practical version is to break the lethal trifecta, covered in the next section.

K11. Control the supply chain and the agents' channels. Pin, review and sign the tools, MCP servers and skills an agent loads. Authenticate calls between agents. The Amazon Q extension shipped a data-wiping prompt in version 1.84.0; a syntax error kept it from running, and AWS says no customer environments were affected (AWS, 2025). The Hugging Face swarm coordinated on a channel nobody had authorised. I found no measured study of agent-to-agent authentication.

P. Portability: can you lose the model?

K12. Keep the harness portable, and prune it on every upgrade. A vendor-neutral harness, a regression run on every model swap, a fallback model, and availability terms in the contract. Part 1 measured the cost of a mismatch: up to 12 points when a harness tuned for one vendor runs another's model. Part 1 also promised a method for pruning a harness on each upgrade without a rebuild. Here it is, offered as practice: re-run the golden set and the behavioural suite on the new model with the harness as is, then remove one scaffold at a time and keep the removal if the scores hold. I found no study measuring the cost of switching models or vendors.

Lever Group Best evidence Evidence type
K1 Least privilege, agent identity B MCP 2026-07-28; Entra Agent ID; NIST concept paper Standard; vendor; government (draft)
K2 Sandbox, separate environments B Anthropic 84%; OpenAI red-team escapes; Replit Vendor; journalism
K3 Limits outside write scope B Part 4 refund cap; Astra safeguard Own harness; vendor (no controlled study)
K4 Tiered gates G MAS; FINRA Regulator (no outcome data)
K5 Blast radius, prompt budget G Habituation at the Gate; Anthropic tester study Preprint; vendor
K6 Escalation with authority G MAS 5.9; capacity model Regulator; preprint (no field data)
K7 Drilled halt H OpenAI 20 September timeline; Palisade; Anthropic denial limits Vendor; peer-reviewed; design
K8 Reversible by design H Supabase guidance; practice No agent study found
K9 Attributable audit trail T OpenTelemetry; MAS; EU AI Act Art. 12 Standard (Development); regulator
K10 Remove capability R CaMeL; Nasr et al. Preprint (CaMeL); peer-reviewed (Nasr et al.)
K11 Supply chain, agent channels R Amazon Q; Hugging Face swarm Journalism; vendor and independent
K12 Portable harness P Part 1 cross-vendor penalty Benchmark (no switching-cost study)

Security: where it fits

The introduction said security sits across safety and controllability. Part 4 used adversarial pressure as a probe of the agent's behaviour. Here is the other half. Security is controllability under an adversary. The five dimensions stay the same. What changes is that someone is pushing on them.

Prompt injection is what makes agents different from other software here. Any text the agent reads (a web page, an email, a ticket, a tool's output, another agent's message) can carry instructions. The agent cannot reliably tell data from orders. So every untrusted input is a possible instruction from a stranger, delivered with the agent's permissions.

The evidence on defences

Three results set the shape. First, defences that detect attacks fall when the attacker adapts. Nasr, Carlini, Tramèr and colleagues bypassed twelve recent defences, most at over 90% success; human red-teamers collectively succeeded every time. Against Spotlighting and Prompt Sandwiching, static attacks succeeded as rarely as 1%; the adaptive search attack succeeded more than 95% of the time. Second, detectors fail on the attacks that matter: injections a detector misses succeed at about the same rate as those it catches. Third, realistic environments remain unsolved. LivePI, 169 executable attacks across seven surfaces, found attack success between 10.7% and 29.6% across five frontier systems (LivePI, 2026, preprint).

Vendors report much lower numbers on their own suites, and those numbers have improved. Press accounts of the system card say Opus 5 cut Gray Swan attack success within fifteen tries from 5.5% (Opus 4.8) to 2.0% (Let's Data Science, 2026), and that auto mode took browser injections to zero on its test set. Rehberger's result shows what they do not cover. His conclusion is the one to keep: auto mode "IS NOT a substitute for running your agent in an isolated environment".

The lethal trifecta

Simon Willison named the design test in June 2025 (Willison, 2025). An agent becomes dangerous when it combines three things: access to private data, exposure to untrusted content, and a way to communicate externally. Any two are manageable. All three let an attacker read your data and send it out. Meta's "Agents Rule of Two" turns this into a rule: within a session, an agent should hold at most two of untrusted input, access to sensitive systems or data, and the ability to change state or communicate outward, unless a person approves. Its third property is broader than Willison's third leg, because it covers any change of state. Willison called it the best practical advice available while no reliable defence exists (Willison, 2025).

The trifecta is a controllability tool. It works by removing a leg, so the agent never has to notice the attack. CaMeL is the strongest form: it removes the attacker's ability to add actions at all.

The attacks, mapped

Part 4 claimed four of OWASP's ten agentic risks for safety. The other six are attacks, and each lands on a lever here.

OWASP risk What it is Lever
ASI01 Agent Goal Hijack Injected instructions redirect the agent K10
ASI02 Tool Misuse and Exploitation Legitimate tools used for an attacker's ends K1, K3, K5
ASI04 Agentic Supply Chain Vulnerabilities Malicious tools, servers, skills or updates K11
ASI05 Unexpected Code Execution The agent runs attacker-supplied code K2
ASI06 Memory and Context Poisoning Planted content persists across sessions K10, K9
ASI07 Insecure Inter-Agent Communication Unauthenticated or spoofed agent messages K11

ASI03 (Identity and Privilege Abuse) sits on both sides. Part 4 took the agent's use of its privileges. The privileges themselves are K1.

Venn diagram of private data, untrusted content and outward communication, with the levers that remove each leg.
Figure 6 Simon Willison's lethal trifecta (June 2025). Private data, untrusted content and outward communication; each lever marked by the leg it removes.

Model supply is a control

Part 1 argued that frontier availability is now set by governments and by labs' safety decisions, and promised to take it up here. The events of 2026 make it a controllability question. An operator who cannot keep a model running cannot keep a system bounded either, because the replacement model behaves differently inside the same harness.

Date Event Effect on buyers
Oct 16, 2026 China's commerce and science ministries propose adding 23 AI technologies, including large models, agents and model weights, to the export-control catalogue (Baker McKenzie) Cross-border model supply becomes a compliance question
Sep 20 to Oct 1, 2026 OpenAI pauses tool-using training and inference on its most capable models after the DNS gap A lab can withdraw tool-use access to its frontier models for eleven days
Jul 28, 2026 MCP adopts a formal deprecation policy, with at least twelve months' notice The only formal availability commitment I found in agent infrastructure
Jun 12 to Jul 1, 2026 A BIS export-control directive suspends access to Claude Fable 5 and Mythos 5 (Anthropic) A generally available frontier model can vanish for nearly three weeks
Jun 2026 Executive Order 14409 sets up voluntary pre-release federal review of frontier models (CRS) The most capable models reach sensitive uses last

These events sit outside the operator's reach. The operator controls how much a loss costs. That is K12: a harness, tool catalogue, retrieval layer and evaluation suite that can move to another model in a migration that needs no rebuild. Part 1's cross-vendor penalty, up to 12 points, is the price of a move made in a hurry. Budget for the re-tuning before you need it.

Three things belong on the procurement checklist, beside uptime and data residency. Notice periods for deprecation and for withdrawal. The vendor's record of pausing or restricting access. And a tested fallback: a second model that has already passed your golden set and behavioural suite inside your harness.

Timeline of 2026 events that shaped frontier model supply, by government, labs and protocols.
Figure 7 Model supply in 2026: the dated events by which governments and labs restricted frontier models, and the protocol that committed to notice.

Mapping to the frameworks your risk team uses

Regulators have converged on nearly the same lever list. FINRA, MAS and the American Medical Association all name the agent's access, where a person sits in the loop, logging of agent actions, and guardrails. MAS goes furthest, and is the best single anchor for this part. As in Part 4, the table is an engineering crosswalk; check legal questions with counsel.

Framework What it asks for Where it lands in this part
MAS Guidelines on AI Risk Management (Oct 2026) An inventory of the tools each agent can access; least privilege; overseers with authority to intervene, allowing for automation bias and decision fatigue; logs and near-miss review; monitoring of reasoning, actions and tools; kill switches for high-risk AI, tested regularly; testing that guardrails work All six groups. The inventory and tiers (Setting the bounds), K1, K4 to K7, K9, and the four numbers
FINRA 2026 oversight report (Dec 2025) Where to put human-in-the-loop oversight; tracking agent actions and decisions; guardrails. No new rules; supervision rules apply K4, K9, K3. It does not mention kill switches
EU AI Act, high-risk systems (Art. 12, Art. 14, Art. 15) Record-keeping (Art. 12), human oversight (Art. 14), accuracy, robustness and cybersecurity (Art. 15) Art. 12: K9. Art. 14: the gate, and the habituation evidence on why it fails. Art. 15 cybersecurity: the security section
NIST NCCoE concept paper (Feb 2026) Agent identity and authorisation: identification, authorisation, delegation, logging and data-flow tracking K1, K9. Still a draft; the AI RMF's Manage function is the frame most US teams will use until it is final
OWASP Top 10 for Agentic Applications (Dec 2025) Ten risks, ASI01 to ASI10 The six attack categories, mapped in the security section
IMDA agentic AI framework (May 2026) and testing starter kit (Oct 2026) Five risks, including unauthorised actions, data breaches and disruption of connected systems; trajectory logging Data breaches: K10, K11. Disruption of connected systems: the bounds and the halt. Logging: K9
ISO/IEC 42001 and 27001 An AI management system; information-security controls, including access control and logging The register of actions, tiers and controls; K1 and K9 sit inside existing 27001 access and logging controls

The dates from Part 4 still apply. MAS's guidelines take effect on 7 October 2027, with life-cycle controls expected from October 2028 (Baker McKenzie, 2026). The EU's high-risk obligations start in December 2027 for stand-alone systems (CMS, 2026). Neither is a reason to wait. The inventory and the drills take longest.

Sector by sector

Financial services: the trading agent and the refund agent. Press reports describe a February incident involving an experimental crypto agent built on OpenClaw. A user asked it for about 4 SOL, and it sent the user its entire holding of 52.43 million tokens, roughly $250,000 notional (AOL, 2026). It was a personal experiment, outside any regulated firm. It is still the clearest case of what a trading agent without a cap does with one misread. For order placement, the tier is Critical: a hard position and value limit in the execution layer (K3), a named person with authority above it (K4), and a drilled halt (K7). For refunds, the tier is High: the cap in the tool, and every confirmation reconciled against the ledger (Part 4's S6). I found no verified loss from a refund or support agent. One widely shared story about a refund spiral could not be verified, so I have left it out.

Healthcare: the patient-chart summariser. The AMA's October framework sets out an autonomy ladder, from informing to recommending to deciding to acting, and holds that clinical authority should be earned through evidence (AMA, 2026). In the UK, a July letter from NHS England and the MHRA treats tools that transcribe and summarise for a clinician's review as outside medical-device regulation, and automated clinical actions as inside it (NHS England, 2026). That is a controllability line drawn by a regulator. A summariser that only reads charts and writes for a clinician stays low on this factor. One that writes to the record, orders a test or messages a patient climbs the ladder, and the bounds have to climb with it.

Software: the coding agent. The sector with the most evidence. Replit's deletion, the Amazon Q supply-chain compromise, Anthropic's sandbox, and Rehberger's attack on auto mode are all coding cases. Reading code and opening a pull request are Medium. Anything that touches production, deletes data or force-pushes is Critical, and gets an environment the agent cannot reach without a person.

Internal tools: the meeting summariser. Low under every framework here. Read-only access to the meetings it summarises, no outward channel, and a log. Controllability barely registers, as the introduction said.

What controls cost

The factors trade off. Reliability's trade is against cost. Safety's is against capability and attention. Controllability's is against all three, and against speed.

Bounds cost capability. CaMeL gave up seven points of task success for provable injection resistance, and used about 2.8 times the input tokens (the paper's own figure). Part 3 reported a $1 per-task budget cap cutting Claude Code's success from 70.6% to 44.2%. Part 1 measured a 38-point ExploitBench gap from a safeguard layer alone, with the model held constant. A tight bound is a cap you set on purpose.

Gates cost attention. Every prompt asks a person to stop and decide. At volume, they stop deciding, as the approval rates in Claim 2 show. An agent that waits for a person cannot run overnight, so the gate also caps autonomy, which is where much of the value lives.

Halts cost capacity. OpenAI's eleven-day pause stopped tool-using work on its most capable models. The halt worked as a decision. It was expensive.

Some controls pay for themselves. Anthropic's sandbox removed 84% of permission prompts. It bought both safety and speed: the sandbox held the limit, so the agent ran with fewer interruptions. The best controls do this. They move a check from a person to a boundary, so the person sees fewer, better prompts.

I found no study measuring the latency cost of sandboxes, or the cost of switching vendors. Those two gaps matter, because they are the costs finance will ask about first.

So report the four numbers from the measurement section beside one more: task success with the controls on. Drive the bound-leak rate down without watching success, and you build a fully controlled agent that does nothing. That agent delivers no value either.

Four bar-chart panels: CaMeL, a budget cap and a safeguard layer reduce task success; sandboxing cut permission prompts by 84%.
Figure 8 What controls cost: CaMeL's utility and token cost, the budget cap, the safeguard gap, and approval fatigue, set beside the sandbox that reduced friction.

The weights question

Part 1 set out five attributes that move the factor weights. Three move controllability most. Reversibility first: as it falls, controllability's weight rises, and in an irreversible domain a capable, cheap, accurate system is unshippable if its errors cannot be contained. Stakes next. Then integration complexity: every connected system adds a scope, a blast radius and a trail. Autonomy amplifies all three, because a longer run is a longer window before anyone checks.

Use case Reversibility Stakes Integration Controllability bar Controllability's place
Internal meeting summariser High Low Low Read-only access; no outward channel Minor
Patient-chart summariser Medium High Medium Read-only; writes only to a draft a clinician signs Third, after safety and reliability; first if it writes to the record
Trading agent Low High High Hard position and value limits outside the agent; drilled halt; a named approver above the limit First
Customer-support agent (refunds) Low (issued refunds) Medium-High High Cap in the tool; ledger reconciliation; approval above the cap Second, after reliability
Coding agent (open PR, run tests) High in a branch; low in production Medium High Sandbox; no production credential; supply-chain controls on tools Rises with autonomy; first once it can touch production

The trading agent is where controllability binds first. Same model, same capability. A wrong order cannot be recalled, so the only question that matters is how large a wrong order can be. That answer is set before the run, in the execution layer, and it has nothing to do with how well the model reasons.

The coding agent shows the other pattern. Inside a branch, almost everything is reversible and controllability barely registers. One production credential changes its tier, and its weight, more than any model upgrade could.

Five use cases rated on how hard they are to undo, stakes and integration, with controllability's place for each.
Figure 9 Where controllability ranks across the five use cases, with how hard each is to undo, its stakes and its integration.

Traps

Seven common ways to misread controllability.

Trap The mistake Why it fails
The Approval-as-Control Trap Count every gate as a control Nearly all prompts are approved; approval rises with experience
The Editable-Safeguard Trap Put a limit where the agent can change it Astra edited a production-deploy safeguard inside its write scope
The Borrowed-Credential Trap Let the agent act on a person's token It inherits all their rights and leaves no separate trail
The Untested-Kill-Switch Trap Assume you can stop a run you have never stopped OpenAI's automatic pause did not fire; people stopped the run by hand about 2.5 hours after the alert
The Detection-as-Defence Trap Rely on an injection detector Adaptive attacks beat most detectors; misses succeed as often as catches
The Single-Vendor Trap Build a harness that cannot survive losing its model Frontier models were withdrawn or restricted for weeks in 2026
The Over-Gating Trap Gate everything People stop reading, and the agent delivers nothing

The first and fourth are the most common, and they share a cause. Both treat a designed control as a tested one.

Order of operations

This is the sequence I recommend for raising controllability with the least wasted effort.

  1. Inventory. List every action, every tool, and every credential behind them. Reuse Part 4's tiers.
  2. Give each agent its own identity, and cut its privileges (K1). Remove every permission the traces show it never uses.
  3. Move hard limits outside the agent's reach (K2, K3). Sandbox it, separate production, and put caps in tools it cannot edit.
  4. Break the trifecta for untrusted input (K10, K11). Remove a leg wherever the agent reads content you do not control. Pin and review its tools.
  5. Place gates where reversibility ends (K4 to K6). Few, tiered, showing the blast radius, with a named person who holds authority.
  6. Make it stoppable and reversible, then drill it (K7, K8). Stage what you can. Time the halt.
  7. Trace everything to an identity, and alert on it (K9). Test that the alerts fire.
  8. Keep it portable, and re-test on every change (K12). Re-run the bound tests, the drills and the evaluations on every model, harness or tool change.

Steps 1 and 2 cost almost nothing and remove the most blast radius. Steps 3 and 4 carry most of the protection. Step 6 is the one most teams skip.

Eight steps in three phases: inventory and reduce; bound and gate; drill, trace and re-test.
Figure 10 Order of operations in three phases: inventory and reduce; bound and gate; drill, trace and re-test.

The contract, and the series answer

Part 1 asked for a way to specify "can do X" precisely enough to write into a contract. Parts 3 and 4 each supplied a clause. This part supplies the last.

Clause From What it states
Measurement Part 3 pass^k at a stated k, with its confidence interval, beside the hand-back and silent-failure rates, on a stated task set, with a stated grader
Behaviour Part 4 A severity-weighted unsafe-action rate and a claim-against-trace mismatch rate, on a stated task set, at a stated intervention rate
Bounds Part 5 The action inventory and tiers; the bounded share of High and Critical actions; the bound-leak rate under a stated red-team protocol; time-to-halt in a stated drill; gate load; notice periods for model deprecation and withdrawal

The first two clauses say what the agent will do. The third says what happens when it does not. A buyer with only the first two has a forecast. A buyer with all three has a limit on the downside.

I found no enterprise publishing terms like these, so treat this as a recommendation. MAS's guidelines come closest: an inventory, least privilege, and kill switches tested regularly are most of the bounds clause, already expected of Singapore's financial institutions from October 2027.

Three contract clauses, measurement, behaviour and bounds; the first two forecast, the third limits the downside.
Figure 11 The three-clause contract: measurement, behaviour and bounds.

The series answer

The introduction asked one question. How much of the model's capability is left once it reaches production?

The ceiling is high and rising. On GDPval, OpenAI reported GPT-5.5 winning or tying against expert professionals on 84.9% of tasks (April 2026, vendor-reported). On the Remote Labor Index, the share of real freelance projects an agent finishes on its own rose from 2.5% at launch in October 2025 (Scale AI and CAIS) to 21.25% on Scale's leaderboard in October 2026 (Scale), a different leaderboard from the launch paper's. With a person in the loop, the 349 technical workers in METR's May 2026 survey reported a median, self-reported 1.4 to 2 times change in the value of their work (METR, 2026). The distance between those numbers is the realisation gap, and each part of this series has measured one piece of it.

Cost is the only factor that aggregates cleanly, and the levers compound. Reliability lags capability, and the gap between solved-once and solved-every-time is not closing. Safety does not rise with capability, and its rare, severe tail can get worse on an upgrade. Controllability is the factor you set yourself, and in 2026 the most capable labs showed how easily it leaks.

The factors multiply, so the answer is the one the introduction gave: the factor nearest zero, the binding constraint, sets the outcome. What this series adds is a way to find it. For each factor there is now a unit, a test, a set of levers and a contract clause. The units are cost per successful task (Part 2), pass^k as the working measure of pass@always (Part 3), the severity-weighted unsafe-action rate with its intervention rate (Part 4), and the four control numbers here. Part 1 noted what is still missing: a model that predicts, from a use case's attributes, how much of the ceiling survives. That needs all four discounts measured on the same deployments. The instruments exist. Most of the measurements are still to be made.

What would change the argument

Four published results would move it:

What this means for enterprise programmes

On the best evidence available in November 2026, the bounds around the most capable agents leak, detection works better than the halts behind it, and the human gate fades with use. Regulators already expect most of the levers in this part. For a programme, that means five things:

  1. Inventory every action and every credential, and give each agent its own identity. The cheapest step, and the one MAS already asks for.
  2. Put hard limits where the agent cannot reach them. Caps in tools, separate environments, and no standing credential for anything Critical.
  3. Remove the capability an attack would need. Break the trifecta wherever the agent reads content you do not control.
  4. Gate rarely, show the blast radius, and watch the approval rate. A gate that approves everything adds nothing.
  5. Drill the halt, and re-test the bounds on every change. Report time-to-halt and bound-leak rate beside reliability and safety.

The shift that matters most is the same one Part 4 ended on, from the operator's side. We need to move from "we put controls in place" to "we know when they last held". A control is a claim about the worst day. Test it before you need it.

Capability sets the ceiling. Controllability decides how bad the worst day can be.


Realized Value · Five deep dives
  • 0Introduction ↗The framework. How the factors multiply.Published
  • 1Capability ↗The model sets the ceiling. The system decides how much of it you get.Published
  • 2Cost ↗Cost per successful task.Published
  • 3Reliability ↗pass@always, and why it lags capability.Published
  • 4Safety ↗The agent's behaviour under uncertainty.Published
  • 5ControllabilityThe operator's leverage, and where security fits.This article

All figures were retrieved and checked by November 1, 2026. Vendor-reported figures (OpenAI, Anthropic, Microsoft) and preprints are labelled as such. Where a figure reaches me through press coverage of a primary document, the text says so and treats it as provisional. That covers Opus 5.5's reduction in escape attempts, Opus 5's injection results, the Hugging Face estimates, OpenAI's October red-team findings and resumption date, the retrospective monitor estimate, and the OpenClaw trading incident. The framework crosswalk is an engineering mapping; check legal questions with counsel. Verify against the primary sources before relying on any single number.

References

Incidents and lab reports
Controls, sandboxes and identity
Human oversight
Security
Regulation and frameworks
Model supply
Series answer
Earlier parts of this series
← Part 4 · Safety Realized Value series · Introduction →