How to set the bar, how to measure, the benchmarks and tools, and the twelve levers in full, behind Part 3.
This page is the reference for Part 3, Reliability is getting it right every time. Part 3 makes the argument in about 25 minutes. This page holds the detail that moved out of it, unchanged. Read it section by section, as you need it.
A section named here that is not on this page is in Part 3. Lever, layer and figure numbers in the text follow this page. Figures were checked on the dates given in Part 3.
Part 1 traced three eras of capability thinking, and Part 2 traced the same three for cost. Reliability went through three as well. Each is defined by the number teams reported.
Accuracy (to mid-2024). One run, one score. Pass@1 on a static benchmark. Variance was noise to average away.
Consistency (mid-2024 to 2025). Sierra's τ-bench (June 2024) introduced passk: run the same task k times and count it only if every run succeeds. It graded the final database state, which a program can check. GPT-4o solved about 61% of retail tasks on one try and under 25% on all eight (Yao et al., 2024). That gap made consistency a number people could argue about.
Operational reliability (2026). Consistency is one dimension among several. The Princeton framework adds robustness to faults and paraphrase, predictability (does the agent know when it will fail), and safety (Rabanser et al., 2026). Benchmarks started reporting passk next to pass@1 (APEX-Agents, τ³-Banking). The question shifted from "how often does it work" to "can I predict when it won't".
The same split decides where a wrong action belongs. Each factor answers a different question about the same incident. Reliability asks whether the agent did what it meant to do, correctly and every time. Safety asks whether what it meant to do was acceptable, and whether it said truthfully what it did. Controllability asks whether the system could stop it, limit it or undo it.
Out-of-scope actions go beyond the task or the authority the agent was given. When the agent chose the action, it is a safety failure: it knew enough to take another path and did not. When the system allowed it, the bounds were too wide, and that is a controllability failure. Most out-of-scope incidents are both.
Unintended actions are outcomes nobody meant. When the agent meant to do the right thing and slipped (a misheard digit, a wrong parameter, a misclick, a flaky tool), it is a reliability failure. VoiceAgentBench shows this pattern: for one model (Gemma3-27B) on Whisper transcripts of non-Hindi Indic languages, the right tool was still chosen 91% of the time, but parameters were right only 41.2% of the time, against 79.6% on correct transcripts (VoiceAgentBench, 2026, preprint). When the instruction was unclear, or the input low-confidence, and the agent acted without asking, the first preventable cause is the choice to act, and that is safety. When the action could not be caught or undone, controllability failed as well.
Figure 2 has one more branch. The agent meant the action, stayed in scope, and the result was still wrong. If it never gets that task right, that is a capability gap (Part 1); if it gets it right only some of the time, it is reliability.
One incident often touches all three. A voice agent that refunds the wrong account failed on reliability (it misheard), on safety (it acted on a low-confidence input without reading the details back) and on controllability (nothing asked for approval before money moved).
To keep the numbers clean, I use one rule. Assign each incident to its first preventable cause (a slip, a choice, or a missing bound) and record the other layers that also failed. That avoids counting one failure twice, the same discipline Part 1 applies when it keeps reliability out of capability. Honest reporting is counted separately: whatever caused the action, a false account of it is a safety failure on its own.
| Incident | Reliability | Safety | Controllability |
|---|---|---|---|
| Replit agent deletes a production database during a code freeze (The Register, 2025) | Not the main cause | Chose a destructive action, then said rollback was impossible | Agent had production write access |
| Voice agent refunds the wrong account | Misheard digit | Acted without reading back a low-confidence input | No approval before the refund |
| GPT-6 Astra edits a production-deploy safeguard (OpenAI, 2026) | Not the main cause | Chose to act outside scope | The safeguard was inside the agent's write scope |
| Support agent tells a customer a $150 refund was processed while the tool refused it (Part 4) | Not the main cause | Reported an action that did not happen | The cap in the tool held |
Bold marks the first preventable cause. The voice-agent row is illustrative.
| Family | The question | Dimensions | How it changes the bar |
|---|---|---|---|
| Consequence | What does one slip cost? | Severity of a wrong result (money, health, legal standing); reversibility; who bears it; reliance (advisory, or acted on without checking) | Sets how high the bar must be: the k you test at, and the silent-failure rate you can accept |
| Exposure | How many chances to slip? | Volume of runs; steps per task and turns per conversation; number of systems touched; variety of inputs (phrasing, accents, call audio, scanned documents, screens); drift in inputs and data; model and tool changes under you | Turns a rate into a count. The more runs and steps, the higher the per-run rate must be |
| Detection | Will you find out? | Verifiability (code check, ledger, expert only); time to detect against time to harm; whether failures are silent or handed back | Can lower the bar when a check catches the failure before it takes effect. A caught failure counts as a hand-back |
| Context | What must you prove, and to whom? | Contracts and SLAs; regulation; who signs off; independent assurance | Scales the evidence: sample size, k, confidence interval, and who runs the test |
Three rules combine them.
Volume turns a rate into a count. Customers and auditors see failures one at a time. To hold failures to one a day on average, an agent that runs 100 times a day needs 99% per run. At 10,000 runs a day it needs 99.99%. The same model, on the same task, can be fine in a pilot and unacceptable at scale.
Labs face the same arithmetic. Anthropic writes that its internal testing and evaluations "operate at a smaller scale than the full deployment" it anticipates, "so behaviors that only emerge rarely or in hard-to-anticipate settings will be difficult for us to catch" (Fable 5.2 system card, §6.1.1). The sentence is about alignment risk. It applies to any rare failure: a pilot is a small sample of production.
Consequence sets the bar, detection discounts it. An irreversible action that nobody checks needs the highest bar. The same action behind a code check that catches the failure first needs a lower one, because the check turns silent failures into hand-backs. A check that only sometimes catches it discounts nothing for the cases it misses.
Re-estimate on every change. Exposure changes with each model, tool or prompt change, and with the data. And temperature 0 still leaves room for variation. Thinking Machines ran the same prompt 1,000 times at temperature 0 on one open model and got 80 different completions. The cause was server load changing the batch size the request ran in (He, 2025). A single run is one sample from a distribution.
Part 4 asked whether a modality makes a wrong action more likely. Reliability asks a different question: does the modality widen the gap between the best run and the worst? The evidence below is mostly about that gap. Where I found none, I say so.
Long conversations. This is the strongest result I found. Laban and colleagues ran 15 models through more than 200,000 simulated conversations. When the same instruction arrived in pieces over several turns, performance fell 39% on average against giving it all at once. They split the fall into two parts. Aptitude, the score of the best runs, fell about 15%. Unreliability, the gap between the 90th and 10th percentile of ten runs, rose 112%. Setting temperature to 0 for both the assistant and the simulated user still left a gap of about 30 points. Their summary: "when LLMs take a wrong turn in a conversation, they get lost and do not recover" (Laban et al., 2025, preprint). Most of the cost of multi-turn work is reliability. A golden set of single-turn questions will not see it.
Voice in. τ-Voice ran the same customer-service tasks as text and as live voice calls. GPT-5 completed 85% in text. Voice agents completed 31% to 51% in clean audio, and 26% to 38% with background noise and varied accents. The accent effect depended on the provider: one lost 18 points on the retail tasks from accents alone, another lost 1 (Ray et al., 2026, preprint). Across languages, Spanish, Brazilian Portuguese and Hindi stayed within about 3 points of English, while Korean fell 14.7 points and Mandarin 8.4 (Ray et al., 2026b, preprint). Neither paper reports passk. Voice also adds a second source of variance before the agent sees anything. In the Careless Whisper study, Whisper's invented phrases "were produced non-deterministically": the same file, transcribed twice, could yield different invented text. About 1% of 13,140 clips contained one, more often for speakers with aphasia (Koenecke et al., 2024). When my team tested speech-to-summary and action-item agents, the failures followed one chain: poor audio and accent variability caused transcription errors, which led to misclassified intent and the wrong outcome (Part 4). Part 4's VoiceAgentBench result shows the usual shape of the slip: the right tool, the wrong parameters (91% against 41%) (VoiceAgentBench, 2026).
Screens. A study of computer-use agents on OSWorld ran each task several times. Claude Sonnet 4.6 solved 70.2% on one try and 61.2% on all three. GPT-5 in the Agent S3 framework went from 57.6% to 45.4%, and Kimi K2.5 from 50.8% to 35.7%. Cosmetic changes to the screen made Claude significantly less consistent. The largest gain came from clarifying the task before execution: GPT-5's pass3 rose from 45.4% to 57.6%. Deterministic decoding helped two models and hurt one (Gonzalez-Pumariega et al., 2026, preprint). That is R6 measured on screens: an agent that resolves ambiguity first is a more consistent agent.
Documents and images. The same content presented as text or as an image of that text can get different answers. A study of 15 multimodal models found the inconsistency persists even when the model reads the text correctly, and that text colour and resolution change the result while font does not (van Sprang et al., 2026, CVPR 2026). For document extraction, I found no published study of run-to-run variance. One September 2026 benchmark evaluated commercial APIs in a single pass. It also gives a useful warning: one fine-tuned model, shown documents unlike its training set, returned the same date for all 400 of them (Patel et al., 2026, preprint). That is consistent and always wrong. Consistency and correctness are separate properties, which is why the always, sometimes and never split needs a golden answer.
Structured output. Typed fields are what make passk checkable by code: compare the fields across runs. Constrained decoding now makes parse failures rare (0% for the commercial systems in the document benchmark above). But format can change the answer. In 2024 tests, strict JSON schemas cut accuracy on maths word problems and raised its spread across prompt variants: for Claude 3 Haiku the standard deviation went from 0.8 points in free text to 22.8 in JSON (Tam et al., 2024, preprint). Newer schema-based APIs closed most of that gap in the same paper's later runs. The practical rule: let the model reason in free text, then emit the fields.
Voice out. A spoken answer can't be re-read, so its slips go unnoticed. The evidence on text-to-speech errors with numbers, dates and codes is thin and vendor-run: one April 2026 benchmark of 31 kinds of non-standard text reported "significant failure rates" across leading providers without publishing independent figures (Async, 2026, vendor). One check you can run yourself: transcribe the agent's own speech and compare the names and numbers with the text it meant to say.
| Modality | Run-to-run evidence | Typical slip | Measure | What helps |
|---|---|---|---|---|
| Typed text, single turn | Thinking Machines: 80 different completions in 1,000 runs at temperature 0 | A different answer to the same question asked another way | passk across paraphrases of each golden item | Paraphrase variants; typed outputs compared field by field |
| Long conversations | Laban et al.: best-to-worst gap +112% when instructions arrive over several turns | An early wrong assumption the agent never recovers from | Spread across 10 runs of the same multi-turn script, with mid-conversation corrections | Restate the full request at checkpoints; test long sessions as well as single turns |
| Voice and call audio | Whisper's invented text varies between runs; no voice-agent passk published | Right tool, wrong name, number or date | Entity and number accuracy per accent group, language and channel, including 8 kHz audio | Pass recogniser confidence on; read back key details; keep the audio |
| Screens | OSWorld: pass1 70.2% to pass3 61.2% for Claude Sonnet 4.6 | The wrong element, or a changed layout | passk with cosmetic changes to the screen | Clarify the task first (R6); state checks and undo (R7) |
| Documents and images | Cross-modal inconsistency in 15 models; no repeated-run extraction study found | A misread field acted on | Field-level agreement across repeated extractions, against a golden answer | Extract twice and compare; show the source crop |
| Structured output | Strict JSON raised spread on reasoning tasks in 2024 tests | Valid format, different content | Field-by-field agreement across runs | Reason in free text, then emit the fields |
| Spoken output | Vendor-run only | A number or code read out wrongly | Transcribe the agent's speech and compare entities | Spell out codes; offer a text copy |
Rows built on preprints or vendor-run evidence are marked in the text. The "what helps" column for documents and spoken output reflects practice; I found no measured results for it.
Two things follow. First, an average over channels hides the worst one. Report passk per channel and per user group, for the same reason Part 4 rates severity per stakeholder. Second, the gaps are themselves a finding. Voice agents are going into contact centres now, and I found no published passk for any of them. If you run one, expect to measure it yourself.
Part 4 maps the frameworks your risk team uses. Two speak to reliability directly. The EU AI Act asks high-risk systems to "perform consistently" in accuracy and robustness "throughout their lifecycle", and to be "as resilient as possible regarding errors, faults or inconsistencies" (Article 15). NIST's AI RMF lists "valid and reliable" as a trustworthiness characteristic (NIST AI 100-1). Singapore's October 2026 starter kit is titled for testing agentic systems "for Safety and Reliability" (IMDA, 2026). How the bar applies, by sector:
Passk is a statistic, and most published versions of it come without error bars. Three things matter.
Sample size. With 100 tasks and a measured passk of 56%, the 95% confidence interval is about ±10 points. With 30 tasks it is about ±18. Two models 10 points apart on a 100-task set may not differ at all. Mercor's APEX-Agents 1.1 post reports no intervals; by the same arithmetic, a score near 56% on its 240 tasks carries about ±6 points. Report yours. Husain and Shankar give the production version of the rule: track confidence intervals on production metrics, and "If the lower bound crosses your threshold, investigate further."
The estimator. Run each task n times, with n at least k, and count c passes. The share of k-run subsets that all pass is C(c,k) / C(n,k). Average it across tasks. This is the estimator τ-bench uses (Yao et al., 2024). Running more than k times gives a steadier estimate than running k times.
Grader errors compound with k. Husain and Shankar measure a judge by two rates against expert labels: the true positive rate (when the expert says pass, how often the judge also says pass) and the true negative rate (when the expert says fail, how often the judge also says fail) (Husain, validate-evaluator). Suppose the true positive rate is 95%, so the grader wrongly fails 5% of correct runs. On one run, that costs five points. On eight runs, an agent that is right every time scores 0.958, about 66% pass8. A third of the score went to grader error. Errors the other way inflate pass@1: a grader with a low true negative rate sometimes passes a wrong answer, which makes never-pass items look flaky. This is why a code check on a golden answer is worth more for reliability than for capability.
Measuring costs money. Two hundred tasks at k = 8 is 1,600 runs. At $2 a run, that is $3,200 for one full pass, before paraphrases, and you re-run it on every change. Budget for it per run, the way Part 2 budgets inference.
A reliability number is only as reliable as the thing that grades it. Five findings from 2026:
Unfinished runs need a rule as well. On Real-World Finance, Fable 5.2 did not finish six of 123 tasks within the three-hour limit. Counting those six as losses cut its lead over Opus 5.5 from about 66 Elo points to about 43 (§8.14.6). On FrontierCode 2.0, 22 of its 660 max-effort attempts timed out. Counted as zero, they take its score from 45.7% to 44.4%, below Opus 5.5 at 45.5% (§8.4). Decide before the run how timeouts and unfinished tasks are scored, and report the count beside the score.
Benchmarks have bugs too. OSWorld 2.0 issued a bug-fix release in September 2026 that changed task snapshots and evaluation workflows (OSWorld-V2 release notes). Deterministic grading helps: τ-bench checks the final database state, and Agents' Last Exam scores tasks with deterministic graders against hidden references (Snorkel, 2026b). Where a code check is possible, use it.
Where it isn't, build the judge the way Husain recommends. One trusted domain expert gives a pass or fail on each output, with a written critique of why. Those critiques become the judge's examples, and the judge is revised until it agrees with the expert (Husain, 2024). His guide to validating a judge says to "Aim for ~50 Pass and ~50 Fail" labelled examples, split into training, development and test sets so the judge isn't tuned to the test (Husain, validate-evaluator). Binary verdicts matter more for reliability than for capability: a 1-to-5 score that drifts by a point between runs makes an inconsistent judge look like an inconsistent agent.
The table at the top of this section lists published results. These benchmarks are public, so a team can run them on its own configuration, k times. Check the grading column first, for the reason above.
| Benchmark | What it tests | Graded by | Reliability reading | Evidence type |
|---|---|---|---|---|
| τ-bench, τ²-bench, τ³-Banking | Agents serving simulated users under a written policy | Final database state | passk built in | Preprint; leaderboard self-reported |
| APEX-Agents | Banking, consulting and legal work across 33 environments | Rubrics, LLM judge | pass@k and passk; public dataset | Vendor-built |
| OSWorld 2.0 | 108 long workflows across desktop applications | Checkpoints in the environment | Full completion and partial credit; run it k times yourself | Independent; preprint |
| Agents' Last Exam | Long professional workflows across 55 fields | Deterministic graders against hidden references | Full pass and partial; repeat runs for passk | Preprint |
| HAL reliability (GAIA, τ-bench airline) | Twelve reliability metrics with paraphrase and injected faults | Harness metrics; five runs per task | Consistency, robustness, predictability; harness is public | Peer-reviewed (ICML 2026) |
Two cautions. Benchmark scores change with the benchmark version, as OSWorld 2.0 showed, so record which release you ran. And a public benchmark tells you whether a model is worth testing. Your golden set tells you whether it is reliable enough to deploy. Husain and Shankar are blunter about generic metrics: they "waste time and create false confidence when you use them as quality measures".
pass_k_{k}, for "the probability that all k epoch attempts succeed", alongside pass_at_{k} for pass@k (Inspect docs). A golden set loaded into Inspect gives passk with one setting.Commercial and in-house harnesses, including the one my team builds for Sunshine.guardian, follow the same pattern: a scenario, a simulated user, repeated runs, a grader. Ask any of them for per-item results across runs. A single aggregate hides the flaky middle.
Public benchmarks tell you about their tasks. To get passk for yours, you need three things:
Anthropic's eval guidance names this pattern: capability evals that "graduate" into regression suites held near 100% (Anthropic, 2026a). It also found infrastructure resource limits alone changed coding-eval success by six points (Anthropic, 2026b). So a harness or infrastructure change needs the regression suite as much as a model upgrade does.
What this looked like in practice. In IMDA's Global AI Assurance Sandbox, my team at NCS tested a career guidance chatbot we built for a client (Part 4 describes the engagement). Consistency was one of the risks we tested. We used conversations of 10 to 25 turns with paraphrases and mid-conversation corrections. Across nine scenarios we ran 45 simulated conversations and aggregated the scores across runs to see how consistent the system was. The scoring was strict: if one criterion failed, the response failed. The review found that the system gave consistent recommendations (AI Verify Foundation, 2026).
R1. Workflow before agent. The cheapest reliability is the variance you never create. Anthropic's guidance is that workflows give predictability and consistency for well-defined tasks, and that many applications need only a well-built single call (Anthropic, 2024). This is Lever 3 in Part 2, and it helps both cost and reliability.
R2. Fewer steps. From the compounding math, every step removed multiplies task success by 1/p. Batch tool calls, merge redundant lookups, shorten the plan.
R3. Deterministic tools and a fixed environment. Structured outputs, typed tool schemas, pinned dependencies. Mercor's 1.1 fix list pre-installed every dependency that appeared 25 or more times across 26,302 trajectories, removing a whole class of run-to-run variance (Mercor, 2026b).
R4. Code verifiers and tests. Where the outcome can be checked by code (a test suite, a database state, a reconciled total), check it on every run. This is what makes τ-bench and Agents' Last Exam trustworthy.
R5. Critic or verifier loops. CAIS added a worker-critic loop on the Remote Labor Index because workers are too optimistic about their own output, and reports it improves results without publishing an ablation (CAIS, 2026). One preprint ablation, on an agent flow for chip accelerator design, found a verifier loop raised cost per success from $5.43 to $7.58 (arXiv 2605.15237). Verification buys reliability, and you pay for it.
A critic can miss what the worker missed. In a biology red-teaming exercise in Anthropic's November 2026 card, graders found overstated claims of efficacy in the agentic submissions. Those claims "were not flagged during the model's review of its own work, nor during review by auxiliary agents" (Fable 5.2 system card, §2.2.2.1).
A verifier is a classifier, so it has the two error rates Part 4 measures for guardrails. A false accept lets a wrong result through as a success: a silent failure. A false reject sends a correct result back for a retry or a hand-back it didn't need, which costs money and attention. Measure both on the golden set, by running the verifier over outputs you already know are right and outputs you know are wrong. A verifier from the same model family as the agent is likely to share its blind spots, so prefer a code check, and mix families where you can't.
R6. Ask before guessing. On APEX-Agents 1.1 (model names as Mercor gives them), Gemini 3.8 Flash searched communications for clarifying information on 93.1% of runs. Kimi K3 and GPT-5.6 Sol did so on about 15%. When Kimi K3 did search, its task-level pass@1 rose 29.7% (Mercor, 2026b). Inconsistent information-seeking is a measurable cause of inconsistent outcomes. R6 is about looking for information that already exists. Asking a person, and checking that the answer is real consent, is S4 in Part 4.
R7. Checkpoints and resumable state. Save state after each completed step so a failure restarts from the last good point.
R8. Idempotent, retry-safe tools. A retried call must not charge a card twice or file a ticket twice. Idempotency keys on every write.
R9. Bounded retries with fallback. OpenAI's guidance is to cap retries and actions, and escalate to a human when the cap is hit (OpenAI, 2025). Every escalation is a hand-back that a person has to handle, so count it against the hand-back budget.
I found no published agent study with numbers for R7 and R8. They are standard practice in distributed systems, and I present them that way.
R10. Abstain on low confidence. Predictability is what makes this lever work. An agent whose confidence tracks its accuracy can be gated: high confidence proceeds, low confidence escalates. HAL's calibration scores show this varies by model and can get worse on an upgrade (Opus 4.7). The gate design belongs to Part 5. The calibration that triggers it belongs here. Part 4 covers the same lever from the safety side, as S5.
R11. Trace-level telemetry. OpenTelemetry's GenAI conventions now define agent and tool spans (invoke_agent, execute_tool), though they remain at Development stability (OpenTelemetry, 2026). Trace every run, so passk can be computed from production runs.
R12. Regression evals on every change. Model upgrade, harness change, tool change, infrastructure change. Re-run the golden set k times on each, and compare the per-item split as well as the average. Part 4's S11 is the behavioural twin of this lever.
| Lever | Group | Best evidence | Evidence type |
|---|---|---|---|
| R1 Workflow before agent | D | Anthropic, Building effective agents | Provider primary |
| R2 Fewer steps | D | Compounding arithmetic | Analytical |
| R3 Deterministic tools, fixed environment | D | Mercor 1.1 dependency fix | Vendor |
| R4 Code verifiers and tests | V | τ-bench, Agents' Last Exam grading | Preprint |
| R5 Critic and verifier loops | V | CAIS RLI; arXiv 2605.15237 | Independent (no ablation); preprint |
| R6 Ask before guessing | V | APEX-Agents 1.1 analysis | Vendor |
| R7 Checkpoints, resumable state | R | Distributed-systems practice | No agent study found |
| R8 Idempotent tools | R | Distributed-systems practice | No agent study found |
| R9 Bounded retries, fallback | R | OpenAI, practical guide to agents | Provider primary |
| R10 Abstain on low confidence | C | HAL calibration metrics | Peer-reviewed |
| R11 Trace telemetry | O | OpenTelemetry GenAI conventions | Standard (Development) |
| R12 Regression evals | O | Anthropic, Demystifying evals; infrastructure noise; IMDA regression sets | Provider primary; government |
Part 1 set out five attributes of a use case that change the factor weights: stakes, autonomy, reversibility, integration complexity and verifiability. Stakes and reversibility set how high the reliability bar must be (the Consequence family above). The other three set how much weight reliability carries against the other factors.
Applied to the same five use cases from Part 1, with the reliability bar each one needs:
| Use case | Autonomy | Integration | Verifiability | Reliability bar | Reliability's place |
|---|---|---|---|---|---|
| Internal meeting summariser | Low | Low | Medium | pass@1 is fine; a human reads every output | Minor |
| Patient-chart summariser | Low | Medium | Low | High consistency; a missed item is an incident | Second, after safety |
| Trading agent | Variable | High | High | passk near 100% on irreversible steps | Second, after controllability |
| Customer-support agent (refunds) | High | High | High | passk at customer volume | First |
| Coding agent (open PR, run tests) | High | High | High | Tests catch most failures; passk matters less per run | Second, after capability |
These are use-case weights. Within a use case, set the bar per action, as in A bar per action above.
The customer-support case is where reliability is the first constraint. Every conversation is a fresh try. A 90% pass@1 agent fails one customer in ten, and the customers who get the failure don't see the average.
This part covers how to measure reliability and how to raise it. Three questions are related to it.
Contract-ready specifications. Part 1 promised a way to write "can do X" into a contract. This part supplies the measurement clause: passk at a stated k, with its confidence interval, beside the hand-back and silent-failure rates, on a stated task set, with a stated grader. Part 4 adds the behaviour clause. The bounds clause (what the agent may do when it is wrong) is Controllability's, in Part 5. I found no enterprise publishing agent SLOs or passk terms yet, so this is a recommendation. One preprint proposes an agent error budget, B = (1 − SLO) × number of decisions, with autonomy that widens or narrows as the budget is used (arXiv 2608.10153). It has not been tested in deployment.
Behaviour under ambiguity. Whether the agent invents an action, hedges or asks is Safety, in Part 4. Princeton's safety dimension is reported there.
Harness portability on upgrades. This part covers the regression eval. Pruning the harness on each upgrade stays in Part 5.
Four published results would change it: