Realized Value Series · Part 2 · Reference
Cost · The eighteen levers · Nov 2026

Cost: the eighteen levers in detail

The published results, tables, caveats and sources behind each of the eighteen cost levers in Part 2.

Sunil Sivadas, Ph.D. Nov 2026 Reference · ~40 min, read by lever Part 2: Cost

This page is the reference for Part 2, Cost engineering for agentic AI. Part 2 makes the argument in about 25 minutes: the unit, a summary of each lever, the order of operations and the traps. This page holds the detail. Read it lever by lever, as you need it.

Three things are here. The first is the eighteen levers, each with its published results, caveats and sources. The second is the evidence behind them: the leaderboards, the table that names the evidence for every lever, and the benchmark lifecycle. The third is the six open problems.

Sections named in quotation marks, such as "What cost is", are in Part 2. Lever numbers and the 5 Cs codes are the same on both pages. Figures were checked on the dates given in Part 2.


The eighteen levers

I group the levers by the 5 Cs. Each lever keeps one number throughout, and the C-code under each heading (for example C2.3) marks its category. Cross-references in this article use the lever numbers.

C1. CONTROL

What work should run at all?

Lever 1. Model routing

C1.1. Route each step to its own tier.

Evidence: Peer-reviewed (RouteLLM, ICLR 2025: 95% of GPT-4 quality at 26% strong-model calls). Provider primary (AWS Bedrock Intelligent Prompt Routing: 63.6% average cost saving at baseline accuracy, self-reported). Production (Uber GenAI Gateway; Cloudflare, 10.57B tokens/day).

Assign a model tier to each step of the agent. Classification and extraction go to a small model. Eligibility checks, judgement and drafting go to a large one. The published results:

Routing by tier: classify the task, send it to a small, mid or frontier tier, check the output, and escalate one tier on failure.
Figure 1 Routing by tier. Classify at task or phase boundaries, send each task to the smallest tier that passes, and escalate one tier on a failed check.

How much routing can be worth. Vals AI estimates that an oracle router would beat the best single model on the Vals Index by 14 points at about 40% lower cost (Vals AI, 14 Oct 2026). An oracle router picks the best model for each task in hindsight. No real router reaches it, so I read the figure as an upper bound. The price ladder sets the other bound. Claude Haiku 5.5 costs 2.5% of Claude Opus 5.5 per token for prompts up to 100,000 tokens. A step moved down that ladder saves up to 97.5%. Above 100,000 tokens, Haiku 5.5 costs five times more, and the saving falls to 87.5% (Anthropic, Haiku 5.5). Long agent trajectories cross that line.

Switching models mid-session has a cost. A new model starts with a cold prompt cache. Vals says it plainly: in long-running sessions, switching to a cheaper model does not automatically bring cost savings. Route at task or phase boundaries, where the cache would be rebuilt anyway. Measure the cache-miss cost before routing step by step.

My working rule is an escalation rate of 5 to 15%. If half or more escalates, the routing is wrong. A common mistake is to "use the small model everywhere". The goal is the smallest acceptable model for each step, with a known escalation path.


Lever 2. Deflection

C1.2. Answer without calling the model.

Evidence: Peer-reviewed (vCache, ICLR 2026: up to 12.5× higher hit rate than GPTCache baselines while holding a user-specified error bound; 57% hit rate at under 0.5% error in one configuration). Provider primary (Azure API Management llm-semantic-cache-lookup policy, preview; AWS ElastiCache semantic cache: up to 86% cost reduction at 91% answer accuracy on 63,796 queries). Production (Walmart: waLLMartCache, ICPR 2024, a distributed multi-tenant cache; hit rate in the table below).

A rule-based layer in front of the model, a small classifier or a response cache can answer a request without calling the model at all. One distinction matters here.

Routing decides which model answers. Deflection decides whether any model answers.

This changes the arithmetic. Routing brings the cost down to the cheap model's price and stops there. When 87% of prompts go to the small model, you still pay the small model on all 87%. Deflection removes the call. A cache hit costs one embedding and one vector lookup, roughly a thousandth of a generation call. A 40% deflection rate removes 40% of inference cost.

Three cost levers compared. Deflection removes the call. Routing sends it to a cheaper model. Prompt caching discounts the input of a call that still runs.
Figure 2 Deflection, routing and prompt caching act at different points in the request path, so their savings multiply.

In the request path, the order is as follows.

Deflection sequence: exact-match cache, then semantic cache with reranking, then deterministic intent handling, and only then a model call.
Figure 3 Deflection order. Exact-match cache first, then semantic cache with reranking, then deterministic intents, and only then a model call.

Realistic deflection rates depend on the workload. The "90%+ hit rate" in vendor material mixes up two things. One is how accurate a match is once a hit occurs. The other is how often hits occur. The published production and benchmark numbers:

Workload Realistic rate Source
Repetitive query sets (benchmark) 62–69% GPT Semantic Cache (arXiv 2411.05276): 61.6–68.8%
Code and other high-repetition categories 40–60% Category-aware caching (arXiv 2510.26835)
E-commerce search (Walmart) About 50% Portkey's summary of a talk by Rohit Chatter, Walmart Global Tech
Predictive instruction cache, first turn 20–51% InstCache (arXiv 2411.13820): 51.3% on LMSys
Open-ended conversational chat 5–15% Category-aware caching
Agentic tool loops Highly variable Cortex (arXiv 2509.17360)

The hit rate follows how often queries repeat. Traffic where a few queries dominate (a power law) caches well. Uniform or volatile traffic does not. Exact repeats alone are a large share of real traffic. That is why a cheap exact-match layer belongs in front of the semantic one.

Check the economics first. A remote vector lookup takes roughly 30 ms. At that latency, a semantic cache needs a hit rate of around 15 to 20% to break even (arXiv 2510.26835). The same paper puts the break-even for in-memory HNSW search at 3 to 5% in its abstract, and its worked examples come out lower. Measure how often your queries repeat before you build.

This lever fails differently from the others. When routing misfires, a cheaper model answers your actual question slightly worse. Quality degrades gradually. When deflection misfires, the system returns an answer computed for a different question, with full confidence. A hit across tenants is a privacy breach.

The published mitigations agree on five points:

  • an exact-match first layer (L1) before the semantic layer (L2)
  • strict namespacing per tenant and per user
  • cross-encoder reranking of the top-K candidates before a match is accepted
  • expiry (TTL) plus invalidation when the source version changes
  • thresholds calibrated per prompt

Never cache personalised, time-sensitive or high-stakes responses. A semantic cache returns a stored answer for a similar request, so that answer can be wrong, out of date or unsafe for the request in hand.

Set the similarity threshold with your risk owners, and treat it as a risk control.


Lever 3. Workflow or agent

C1.3. This choice comes before every other lever.

Evidence: Peer-reviewed (HAL, ICLR 2026, arXiv 2510.11977: harness choice changes the cost-accuracy Pareto position). Provider primary (Anthropic, "Building effective agents"). Benchmark (Hugging Face smolagents: 30% fewer steps, so 30% fewer LLM calls). No production case.

If the task has low cognitive complexity and a stable structure, a deterministic workflow with one LLM call beats an agent loop. Anthropic's engineering essay "Building effective agents" (Dec 2024) draws this distinction. It recommends starting with the simplest workflow that fits.

The Frontier Trap. Part 1 gives it this name: defaulting to the most expensive frontier model for every task. It is the same mistake as building an agent where a workflow fits, one level down. The October 2026 data shows what it costs. On the official Terminal-Bench 4.0 board, GPT-6.1 Sol (in Codex) ties GPT-6 Astra at 58.2%. A full run costs $0.6k against $3.3k (Terminal-Bench 4.0 leaderboard). On Artificial Analysis's Intelligence Index v5, GPT-6.1 Sol scores within 4 points of Claude Opus 5.5 at about an eighth of the cost per task (Artificial Analysis, 27 Oct 2026). The November data repeats it one tier up. On Vals AI's board, Claude Fable 5.2 ties Claude Opus 5.5 at 65.15% and costs more per test ($14.56 against $12.04), at twice the list price per token ($8/$40 against $4/$20). In Anthropic's own run in Claude Code, Fable 5.2 scored 67.6% against 66.4%, inside the standard error (system card §8.5, vendor-reported). On agentic coding in a terminal, the dearer model buys no measurable gain. On the Remote Labor Index it buys a large one: 41.3% of freelance projects against 21.25% (run by Scale AI; system card §8.14.7). Choose per workload.

The Small Model Fallacy is the inverse. A model that is cheap per token can be expensive per success. It may fail often, or spend many tokens to finish. Gemini 3.8 Flash is the example on Terminal-Bench 4.0 (see "What cost is"). On the official board, Claude Sonnet 5.5 at max effort cost $7.3k for a full run against $4.7k for Opus 5.5, with a lower score. It used 19.4 billion tokens against 8.0 billion. The rule is the same in both directions. Choose by cost per successful task on your own workload.

Decision tree for choosing a deterministic workflow or an open-ended agent loop.
Figure 4 Choosing between a deterministic workflow and an open-ended agent loop. Cognitive complexity and the stability of the structure decide.

Lever 4. Confidence-gated work

C1.4. Pay for extra checks only when a cheap signal asks for them.

Evidence: Peer-reviewed (FrugalGPT, Chen, Zaharia and Zou, TMLR: matches GPT-4 at up to 98% lower cost). Preprint (UCCI, arXiv 2605.18796: 31% lower normalised inference cost than a large-model-only baseline, calibration error 0.12 to 0.03). Provider primary (AWS Bedrock IPR escalation). Production, with a different mechanism (Cloudflare AI code review: rule-based risk tiers, $0.20 against $1.68 per review, per ZenML's summary of Cloudflare's post). See the caveat on the mechanism.

Running a critic on every output doubles the cost. Running it on the 10% with low confidence costs a tenth as much, and catches most errors when the confidence signal is well calibrated. This is Lever 1's escalation, applied more widely. Pay for verification, expansion or a larger-model pass only when a cheap signal calls for it.

Evidence caveat. Production data on the cost effect of confidence gating is scarce. The pattern appears in the escalation tiers of routing frameworks (RouteLLM, Bedrock IPR). It also appears in outcome-based pricing (for example, per-resolution pricing for customer-service agents). But no named deployment has published a measured cost change from confidence gating alone. I read the savings range as showing the direction of the effect only.


C2. COMPACT

How much information per call?

Lever 5. Context engineering

C2.1. Put only what each step needs into the prompt.

Evidence: Peer-reviewed (AgentDiet, FSE 2026: 40-60% input-token cut at unchanged pass rate; MemGPT, arXiv 2310.08560). Provider primary (Anthropic, "Effective context engineering for AI agents"). Production (Manus: input-to-output token ratio around 100:1, Manus, Jul 2025). Benchmark (OpenHands condensation: under half the per-turn cost, with 54% of a SWE-bench Verified subset solved against 53%, OpenHands, Apr 2025).

Anthropic's engineering guide from September 2025 is now the standard reference: "Effective context engineering for AI agents". Practitioners moved from prompt engineering to context engineering because they traced most production agent failures to the context the model was given.

2026 anchor: trajectory accumulation is the largest agent cost driver, and most of it can be removed. AgentDiet (PACMSE FSE 2026, arXiv 2509.23586) reports a 39.9 to 59.7% cut in input tokens and a 21.1 to 35.9% cut in total cost. Quality did not drop. That held across two LLMs and two coding benchmarks. The paper adds OpenRouter telemetry. Of Claude 4 Sonnet's roughly 100B daily tokens, 99% were trajectory input and 1% was generated output. In any non-trivial agent run, the largest cost line is the conversation history that builds up across steps. The answer the model produces is a small share.

The tactics that work in practice:


Lever 6. Data quality and retrieval engineering

C2.2. Good retrieval keeps the context small.

Evidence: Peer-reviewed (PLAID, Santhanam et al., CIKM 2022; BrowseComp-Plus, arXiv 2508.06600). Provider primary (Anthropic Contextual Retrieval: retrieval-failure rate down 49%, 67% with reranking). Evaluation (Sourcegraph, long context against RAG). No production case.

Bad retrieval forces a larger context, a higher top_k, more reasoning to disambiguate, and more retries.

The source data matters as much as the retrieval. Sources that are undocumented, duplicated or out of date raise the same costs before retrieval starts.

The benchmark to beat is Anthropic's Contextual Retrieval (Sep 2024). It reduces the top-20-chunk retrieval failure rate by:

Source: Anthropic. Introducing Contextual Retrieval. Anthropic puts the one-time cost of generating the contextualised chunks at about $1.02 per million document tokens, with prompt caching.

The source data. The title of this lever names data quality, and the evidence above is about retrieval. Two results show the effect of the data itself. In BEAVER, a benchmark built from three private company data warehouses, each database averages about 100 tables and 870 columns, with cryptic names and thin documentation. An agent has to read or search all of that before it can answer, and the best one answered 11.4% of questions correctly (BEAVER, 2026, a preprint). Each failed attempt is a paid run, so cost per successful task rises as the success rate falls. On HOH, outdated passages in the retrieved context caused a performance drop of at least 20% (Ouyang et al., 2025, peer-reviewed).

So the cheapest tokens are the ones a documented source makes unnecessary: a description of each table and field, one authoritative copy of each document, and a date on each. Preparing the data is a fixed cost, and Part 6 counts it there. I found no study that sets the cost of preparing data against the inference cost it saves. That is a gap, and the evidence verdict for this lever stays at Adequate. The introduction treats data readiness as part of the organisation around the agent.


Lever 7. Caching

C2.3. In my reading, the lever with the largest single effect.

Evidence: Peer-reviewed (PagedAttention, SOSP 2023; SGLang RadixAttention, NeurIPS 2024, up to 6.4× throughput on prefix-sharing; Prompt Cache, MLSys 2024). Provider primary (Anthropic: up to 90% cost and 85% latency reduction). Named customer (Notion, Simon Last). Production (ProjectDiscovery, 59% cost cut).

A prompt in three segments: a cached system and tools prefix, a cached per-tenant block, and an uncached per-call tail.
Figure 5 Prompt cache structure. A stable system and tools prefix, a per-tenant block, and a per-call tail that is never cached.

Prompt and prefix caching (at the provider):

Semantic caching belongs to Lever 2. The distinction is worth keeping. Prompt caching and KV reuse make a call that still happens cheaper. Semantic caching stops the call from happening. The mechanism, the limit on savings and the failure mode all differ.

Five caches, each with its own invalidation rule:

Cache type What it stores Invalidation trigger TTL guidance
Prompt/prefix System prompts, tool schemas Prompt version change 5 min to 1 hr
Semantic Query to response Policy or corpus update Hours to days
Embedding Chunk to vector Source change Until source changes
Tool result Stable lookups (holidays, org chart) Source system update Hours to days
Session Parsed context, extracted fields Session end Session duration

Caching without invalidation rules serves stale answers. Track the cache hit rate and the stale-serve rate.

Order the prompt for cache hits. ProjectDiscovery's biggest gain, the jump in hit rate above, came from moving dynamic content (working memory, runtime context) out of the cacheable prefix.

Three changes to cache arithmetic in 2026:

  1. Cache reads got cheaper again in September 2026. Cache reads now cost 0.05× the input price on Claude Opus 5.5 ($0.20 against $4), on Sonnet 5.5, and on GPT-6.1 Sol ($0.10 against $2). Haiku 5.5 and GPT-6 Astra stay at about 0.1×. Claude Fable 5.2, released in November, reads at 0.025× ($0.20 against $8; system card §8.14.5). Gemini 4 Argon's cached input is 0.05× during its introductory pricing. DeepSeek's cache hits cost 2% of a miss (Anthropic; OpenAI; Google; DeepSeek). Cache writes still cost more than uncached input: 1.25× for a short expiry at both Anthropic and OpenAI. On OpenAI's GPT-6 family, changing the reasoning effort on the request breaks the cached prefix. The configuration_update input item changes effort mid-conversation without breaking it (OpenAI, GPT-6.1 Sol guide).

  2. Anthropic isolates caches per workspace (effective 5 Feb 2026). Prompt caches are now isolated per workspace on the Claude API, Claude Platform on AWS, and Microsoft Foundry (beta). Bedrock and Vertex AI remain isolated per organisation (Claude API Docs, 2026). Teams with several workspaces need to revisit their cache strategy. A shared prompt that hit 90% across workspaces now misses entirely across workspace boundaries.

  3. The Opus 4.7 tokeniser produces more tokens for the same text. Anthropic's migration guide, as reported by The Decoder, puts the increase at 1.0 to 1.35× against Opus 4.6. I could not open Anthropic's page to confirm. The per-token list price did not change, so the cost per request can rise by up to 35% on an identical workload. Forecasting tools that assume the same tokeniser across model upgrades will under-report.

One more number is often misread: the vendor "90%+ hit rate" for semantic caching (Lever 2). I treat it as a match-accuracy figure unless the test corpus and the similarity threshold are disclosed.


Lever 8. Retrieval economics

C2.4. Retrieve less, and store vectors more cheaply.

Evidence: Peer-reviewed (PLAID, CIKM 2022: 7× GPU and 45× CPU latency reduction at no quality loss). Provider primary (Qdrant Binary Quantization: 32× memory reduction, up to 40× faster search, 98% recall@50 with oversampling). Leaderboard (MTEB / MMTEB, BEIR). No production case.

Most teams set top_k=10 and never tune it. Reduce it to the smallest k that holds recall on your evaluation set. Use tiered retrieval: a cheap recall pass, then a small reranker, and only the top three to five chunks enter the LLM context. Consolidate related sub-questions into one query.

Together with Lever 6, this is where the 67% drop in retrieval failures from Anthropic's Contextual Retrieval turns into cost savings. There are fewer retries, and a smaller context holds the same answer quality.

Quantisation is the concrete cost lever here. Qdrant's published Binary Quantization results report a 32× memory reduction and up to 40× faster search, holding 98% recall@50. That is with oversampling and rescoring on 4096-dimension embeddings. PLAID (Santhanam et al., CIKM 2022) reports 7× GPU and 45× CPU latency reduction on late-interaction retrieval at no quality loss. Some of the saving attributed to this lever overlaps with Lever 6 (retrieval engineering) and Lever 7 (caching). Count it once.


Lever 9. Output shaping

C2.5. Output tokens are the expensive ones.

Evidence: Peer-reviewed (XGrammar, MLSys 2025: up to 3× on JSON Schema, over 100× on context-free grammars). Provider primary (OpenAI Structured Outputs: 100% schema adherence against under 40%). Benchmark (Aider unified diffs: 20% to 61% on Aider's laziness benchmark, roughly 3× fewer wasted output tokens). No production case.

Output tokens cost four to five times more than input on current models (Claude Opus 5.5: $4 in and $20 out per million tokens; GPT-6.1 Sol: $2 in and $10 out). Reasoning tokens bill as output. Force structured outputs with JSON schemas, hard length caps and enum constraints. Do not allow a narrative "analysis" before the structured answer.


Lever 10. Prompt compression

C2.6. Shrink the tokens, keep the information.

Evidence: Peer-reviewed (LLMLingua, EMNLP 2023: up to 20× compression at minimal loss; LongLLMLingua, ACL 2024: +21.4% RAG performance at a quarter of the tokens). Benchmark-grade only: no named production deployment. See the caveat.

This is different from context engineering. Context engineering chooses which information enters the prompt. Prompt compression shrinks the tokens that carry that information. The two stack.

The LLMLingua family (Microsoft Research) is the most mature tool. It has three variants:

Evidence caveat. LLMLingua's benchmark numbers are peer-reviewed (EMNLP 2023). No named team has published a production deployment with measured before-and-after cost. I treat this as benchmark-grade evidence until a named production case appears.

Where it works well:

Where it fails:

It stacks with caching. Cache the compressed prompt. Hit rates stay similar, and the cached content is already compressed, so the savings multiply.

In operation. Compress the long context block only. Keep the schema instruction and the format specification verbatim. Check output quality on a regression set before rolling out. Compression that breaks parsing costs more than the tokens it saves.


C3. CONSTRAIN

How does the runtime behave?

Lever 11. Harness engineering

C3.1. Bound what the agent can spend.

Evidence: Peer-reviewed (HAL, ICLR 2026, arXiv 2510.11977: 21,730 rollouts, with the harness treated as a cost variable in its own right). Provider primary (Anthropic harness-design guidance). Leaderboard (Terminal-Bench 4.0, SWE-bench Pro, GAIA). Vendor-run benchmarks (LangChain Deep Agents: +13.7 points on Terminal-Bench 2.0 with the model held fixed; the NVIDIA and LangChain Nemotron playbook: about 10× lower cost per run at near-parity accuracy). No production case.

Most hidden cost is in unbounded ReAct loops. They spend tokens on proving they are stuck. Princeton's HAL (arXiv 2510.11977) finds the most expensive model costing an order of magnitude more than mid-tier options, and a 9× cost gap between two agents on one benchmark.

I treat three controls as mandatory:

2026 update: harness engineering became a named discipline. Through 2025 the runtime around the model was called the scaffold, the agent loop, or nothing at all. In 2026 it got a name, a formula and a literature. The working definition is agent = model + harness. The harness is every piece of code, configuration and execution logic outside the model. LangChain's Vivek Trivedy put it as "if you're not the model, you're the harness" (Mar 2026). Addy Osmani set out the thesis in "Agent Harness Engineering" (O'Reilly Radar, Apr 2026). In his words, "A decent model with a great harness beats a great model with a bad harness." Anthropic published harness-design guidance for long-running work. OpenAI adopted the term for Codex. Lilian Weng wrote a Lil'Log post on it (Jul 2026). The named sub-patterns are now stable:

  • self-verification loops
  • context middleware
  • environment-awareness injection
  • loop detection
  • subagent delegation
  • compaction
  • trace-driven optimisation

Why this matters for cost as well as quality: the first hard cost figure arrived in July 2026. NVIDIA and LangChain tuned only the harness around Nemotron 3 Ultra. It reached 0.86 on the Deep Agents suite against 0.87 for Claude Opus 4.8, at about 10× lower cost per run ($4.48 against $43.48), at comparable latency (LangChain; NVIDIA). I read that number as directional. It is vendor-run, on one suite of 127 examples, with the harness tuned for that model. No independent replication has been reported. The direction is still the argument of this lever.

What the leaderboards show. "A smaller model in a good harness beats a larger model in a poor one" used to be intuition. In 2025 and 2026 it became measurable. Four examples:

1. Terminal-Bench 2.0: same model, harness rebuild, +13.7 points. LangChain's coding agent went from outside the top 30 to the top 5 on Terminal-Bench 2.0, from 52.8% to 66.5%. It changed only the harness and held the model fixed at GPT-5.2-Codex (LangChain, 17 Feb 2026).

Caveat on SWE-bench Verified. OpenAI stopped reporting SWE-bench Verified scores in February 2026. It cited contamination and the outsized effect of the agent harness. Its audit found that 59.4% of the hardest unsolved Verified problems had flawed test cases (OpenAI, Feb 2026). The same Opus 4.5 model scores 80.9% on Verified but 45.9% on Scale AI's SEAL-graded SWE-bench Pro under a standardised harness. SWE-bench Pro and SWE-bench Live are the preferred successors. I report SWE-bench Verified results where the cited research used them, because they were the standard at the time. Read them with the contamination problem in mind.

2. Mini-SWE-Agent: a minimal harness does well. SWE-agent's Mini-SWE-Agent is about 100 lines of Python, with no tool beyond bash. Its README reports more than 74% on SWE-bench Verified, ahead of most heavily engineered frameworks.

3. BrowseComp-Plus: retrieval engineering beats a model upgrade. Pairing GPT-5 with the Qwen3-Embedding-8B retriever raised accuracy from 55.9% to 70.1%, with fewer search calls. That is a 14-point gain from changing only the retriever (Chen et al., arXiv 2508.06600, Aug 2025). In the same paper, open-source Search-R1 with a BM25 retriever scores 3.86%. For deep-research agents, the retriever decides much of the outcome.

4. GAIA: the harness outweighs the base model. Hugging Face's smolagents code-agent variant scored 44.2% on GAIA validation and 33.3% on test. GPT-4 Turbo alone scores about 7% (Hugging Face blog). On this benchmark the harness contributes several times what the base model does.

The pattern holds across benchmarks:

Benchmark Same-model harness gap Smaller-model-beats-larger evidence
Terminal-Bench 2.0 / 4.0 +13.7pp (LangChain rebuild, 2.0); within about 2pp at the flagship tier on 4.0 GPT-6.1 Sol ties GPT-6 Astra in Codex at 18% of the run cost
BrowseComp-Plus 55.9% to 70.1% via retriever swap Retriever choice > model size
GAIA smolagents code-agent 44.2% vs about 7% base model Tool-augmented small models beat larger unaugmented ones

Part 1 charts the current same-model harness data (Part 1, Figure 4). The table below uses Terminal-Bench 4.0 in October 2026. It compares each model in its vendor's harness with the same model in a 100-line minimal harness.

Tier Vendor harness against minimal harness Examples
Flagship models Within about 2 points, and sometimes behind Claude Opus 5.5 −0.4; GPT-6 Astra −1.4
Cheaper tiers and weaker labs +3 to +9 points GPT-6.1 Sol +3.2; GPT-6 Sol +5.0; Grok 4.7 +8.8
Cross-vendor pairing Up to −12 points Qwen3.8-Max in Claude Code −11.9
Optimised harness, model fixed +13.7 points LangChain Deep Agents, Terminal-Bench 2.0

So the cost implication now has two tiers. Below the flagship tier, and in any multi-vendor setup, harness work still buys capability that would otherwise cost a model upgrade. At the flagship tier, an off-the-shelf harness no longer adds measurable capability. There, the harness's value lies in what it controls: steps, tokens, caching, routing and permissions. An optimised harness still adds double digits at any tier. The cost lever in this section is the second kind. It bounds what the agent spends, whatever the model.

The Terminal-Bench team built Terminus 2 for this reason. It is a neutral harness with only a headless terminal, used as a testbed for comparing models. Headline numbers on any agent leaderboard are joint scores of model and harness.

Bottom line. When you read "Model X scores Y% on benchmark Z", ask three things. Which harness ran it? At what effort? On which benchmark version? Below the flagship tier, a harness change still changes results by several points. Across vendors it can cost twelve.

November 2026: one harness change raised the newest flagship's score by six points. On ProgramBench, a long-context coding benchmark, Claude Fable 5.2 scored 85.6% in the default harness, behind Opus 5.5 at 91.2%. It ran out of context on 68% of tasks. When the harness showed the context remaining after each step, it scored 91.6% (system card §8.10.1, vendor-reported). This is environment-awareness injection, one of the sub-patterns above. The change added six points at no change in model price.

Leaderboard status, October 2026. Terminal-Bench 4.0 replaced 2.x as the main harness board in August. It has 66 tasks, each about four hours of expert work, with an eight-hour agent budget. Its scores do not compare with 2.x. Epoch AI rated it "Flawed", with scoring defects in 30 of 66 tasks. Read its numbers as the best signal available, with known defects (Part 1 has the detail). On Vals AI's 11 November board, Claude Fable 5.2 and Claude Opus 5.5 tie for first at 65.15%.


Lever 12. Tool design and call discipline

C3.2. Keep tools few, schemas short and outputs compact.

Evidence: Peer-reviewed (Toolformer, NeurIPS 2023; Gorilla, NeurIPS 2024). Provider primary (Anthropic, "Writing effective tools for AI agents": concise response formats cut token usage by up to 65%, from 206 tokens to 72; "Code execution with MCP"). Leaderboard (BFCL v4, τ-bench / τ2-bench). No production case.

Tool calls are network round trips, with their own latency and retry costs. Tool schemas are part of the input tokens on every call. An oversized tool set causes "context confusion". The model invents parameters or picks the wrong tool (Anthropic, Sep 2025).

The controls:

The MCP metadata tax. Model Context Protocol tool registrations are expensive in context. With a large tool catalogue, schema metadata can take up a large share of the context window. Two mitigations work in practice. Discover tools dynamically, as the session needs them. Or move to CLI-first or Skills-based designs, where tool capability lives in code the agent calls and the model reads fewer schemas. If you have a long tool list and a slow, expensive agent, look here first.


Lever 13. Reasoning discipline

C3.3. Set the effort level per workload.

Evidence: Peer-reviewed (s1, EMNLP 2025: budget forcing gives a measured curve of accuracy against thinking tokens; HAL, ICLR 2026: higher reasoning effort gave equal or lower accuracy in 21 of 36 model, agent and benchmark combinations). Provider primary (Anthropic and OpenAI effort controls; Opus 5.5 rejects fixed thinking budgets). Leaderboard (Aider Polyglot, cost-tracked). No production case. See the Price Reversal caveat.

Reasoning tokens are billed at output rates. They are easy to miss in cost forecasts because they do not appear in the response. The rule I follow: don't reason about things you can look up. Current flagship models decide their own reasoning depth within an effort level you set. The application still decides when a reasoning model is the right tool at all. For retrieval and classification, it is not.

2026 update: effort settings replaced thinking budgets, and they change cost more than model choice does. Princeton's HAL (arXiv 2510.11977, ICLR 2026) ran 21,730 rollouts. Higher reasoning effort gave equal or lower accuracy in 21 of 36 model, agent and benchmark combinations. The providers have since removed the fixed budget as a control:

  • Anthropic. On Claude Opus 5.5, thinking is always on. A request that disables thinking or sets budget_tokens returns a 400 error. Depth is set with the effort parameter. The default is medium on Opus 5.5 and high on Sonnet 5.5 and Fable 5.1. Anthropic notes that at the same effort, Opus 5.5 "tends to think more per turn" than Opus 5. It tells developers to re-run their effort sweep after upgrading (Anthropic, What's new in Opus 5.5). Claude Fable 5.2 is available only with thinking enabled (system card, p. 59).
  • OpenAI. GPT-6.1 Sol and GPT-6 Astra support low to max effort and do not support none. GPT-6.1 Terra, GPT-6 Sol and GPT-6 Luna do. A configuration_update input item changes effort mid-conversation without breaking the prompt cache (OpenAI).
  • Google. Output prices include thinking tokens (Gemini API pricing).

The cost range inside one model is now wider than many model swaps. Claude Opus 5.5 scores 35 on Artificial Analysis's Intelligence Index at low effort and 58 at max. Max effort costs 10× as much per task ($0.98 against $10.04) (Artificial Analysis, 27 Oct 2026). Claude Fable 5.2 has a similar range: 53.9 at $3.10 per task at low effort and 64.6 at $26.54 at max, about 9× (Artificial Analysis data via LLM Frontier, 13 Nov 2026).

November 2026: the top effort setting often adds little, and sometimes costs points. Anthropic's system card for Claude Fable 5.2 reports score and cost at each effort level on FrontierCode 2.0, a coding benchmark that Cognition built and ran. Artificial Analysis measured the same model independently.

Effort (Fable 5.2) FrontierCode 2.0 score Cost per task Intelligence Index Cost per index task
Low 34.2% $2.10 53.9 $3.10
Medium 37.6% $2.48 56.4 $3.80
High 45.3% $6.72 61.8 $9.08
Extra-high 49.1% $14.19 62.2 $14.83
Max 45.7% $44.70 64.6 $26.54

Sources: system card §8.4 for FrontierCode 2.0 (run by Cognition, single results with no uncertainty interval); Artificial Analysis data via LLM Frontier, 13 Nov 2026, for the index.

On FrontierCode 2.0, max effort costs 3.2 times extra-high effort and scores lower. Counting its 22 timed-out attempts as failures lowers it further, to 44.4%. High effort (45.3% at $6.72) scores level with Opus 5.5's best (45.5% at $12.82) at about half the cost. On the Intelligence Index, max effort costs 2.9 times high effort for 2.8 more points. Other results in the card agree. On CursorBench 4.0, run by Cursor, max effort used about 353,000 output tokens a task against about 99,000 at high, for 3.1 more points (system card §8.8). On GDPval-AA, extra-high effort used about 57% fewer output tokens than max and scored 1751 against 1831 (system card §8.14.3). On PhysicianBench, the levels from high upward are statistically indistinguishable (system card §8.16.3).

In practice: run an effort sweep per workload, start from the default, and raise effort only on the turns that need it. Use max effort where a measurement on your own tasks justifies it.


C4. COMMODITISE

What does each unit cost?

Lever 14. Fine-tuning and distillation

C4.1. Make a small model good at one narrow task.

Evidence: Peer-reviewed (QLoRA, NeurIPS 2023 oral; LoRA, ICLR 2022). Provider primary (AWS Bedrock Model Distillation). Named customer (Predibase and Convirza, Giuseppe Romagnuolo: 10× lower operational cost, +8% F1, +80% throughput; vendor case study, no longer online). Production (Together AI: a 2 to 3× cost cut at 77% to 87% accuracy; Databricks: 90% accuracy at 5× lower cost).

On a narrow, repetitive task, a fine-tuned small model can match a large model at a fraction of the cost. Published results:

It is worth doing when three things hold. Task volume justifies the dataset and evaluation work. The task is stable. And you have evaluations that will catch a regression.


Lever 15. Service tiers and time-shifting

C4.2. Run each workload in the cheapest tier that meets its deadline.

Evidence: Peer-reviewed (Orca, OSDI 2022: 36.9× throughput at fixed latency, the mechanism behind async discounts; PagedAttention, SOSP 2023). Provider primary (Batch and Flex at 50% across OpenAI, Anthropic and Google; DeepSeek off-peak at 50%; Fast and Ultrafast at 2× and 6×). No production case.

When inference runs is now a pricing axis, alongside what runs. The major providers offer three or four service tiers that trade latency for cost. Map every workload to the cheapest tier that meets its service level.

Provider service tiers in 2026:

Tier Price vs Standard Where it exists (October 2026) When to use
Ultrafast 6× OpenAI GPT-6 Astra ($60/$300) and GPT-6.1 Sol Latency-critical paths that justify any price
Fast / Priority About 2× OpenAI Fast (renamed from Priority on 30 July 2026); Anthropic fast mode on Opus 5.5, $8/$40, up to 2.5× output speed; Google Priority, 1.8× User-facing real time, voice
Standard 1× All providers Default for interactive workloads
Flex 0.5× OpenAI Async user features, non-critical paths
Batch 0.5× (0.6× at Moonshot) OpenAI, Anthropic, Google, Moonshot Bulk jobs, evaluations, nightly enrichment
Off-peak 0.5× of peak DeepSeek: peak is 01:00 to 04:00 and 06:00 to 10:00 UTC on weekdays; all other hours are off-peak Anything that can wait for the window

Source: provider pricing pages, checked 31 October 2026 (OpenAI; Anthropic fast mode; Google; DeepSeek; Moonshot). Anthropic's fast mode cannot be combined with Batch.

Decision rule. Is a user waiting in real time? Use Standard or Priority. Can the result wait a few minutes? Use Flex. Within a day? Use Batch. The same workload in the wrong tier can cost twice as much.

Stacking discounts. Batch combines with prompt caching. On GPT-6.1 Sol, a cached input token in a Batch job costs $0.05 per million, against $2 for an uncached Standard token. That is 2.5% of base (OpenAI pricing). Output tokens get only the Batch half, so output-heavy work saves less.

What qualifies for time-shifting. Anything whose result no user is waiting for. Candidates:

Operational discipline. Track tier drift. Workloads move up the tier ladder when teams do not measure. Schedule heavy Flex jobs off-peak to reduce contention. Build retry with backoff for Flex 429 errors. Tag every request with its intended tier in observability.

Prices now move in different directions by vendor, and some rises are already scheduled. Between September and November 2026:

  • Anthropic cut prices twice. Claude Opus 5.5 is $4/$20 per million tokens, down from $5/$25 for Opus 5. Claude Fable 5.2, released on 9 November, is $8/$40, down from $10/$50 for Fable 5.1 (system card §8.14.5).
  • OpenAI added a tier above its old flagship. GPT-6 Astra is $10/$50, with Ultrafast at $60/$300. GPT-6.1 Sol is at $2/$10 and GPT-6.1 Terra at $1/$5.
  • Google launched at introductory prices with dated increases. Gemini 4 Argon doubles from $2/$10 to $4/$20 on 1 December 2026. Gemini 3.8 Flash doubles from $0.75/$3.75 to $1.50/$7.50 on 1 January 2027. Batch and Priority prices double on the same dates.
  • DeepSeek introduced time-of-day pricing on 16 August 2026. Off-peak costs half the peak rate.

Sources: provider pricing pages, checked 31 October 2026, and Anthropic's system card for Fable 5.2. I draw three implications. Budget at the post-promotion price. Do not forecast that tokens get cheaper every quarter: the price of reaching a given capability falls, but the price of the top model has gone up. And the ladder from the cheapest to the most expensive tier at OpenAI is now about 100× wide ($0.10 to $10 input). That makes Lever 1 worth more than it was a year ago.


Lever 16. Self-hosted vs managed APIs

C4.3. Own the serving stack only when utilisation pays for it.

Evidence: Peer-reviewed (DistServe, OSDI 2024: 7.4× more requests or 12.6× tighter SLO; Orca, OSDI 2022; PagedAttention, SOSP 2023). Provider primary (NVIDIA cost-per-token TCO framing). No production break-even case: Uber's open-weight serving on Ray and vLLM shows the pattern only. No audited public break-even threshold holds up under source review: see the caveat.

The most common question on this lever: when does self-hosting beat paying per token?

Short answer. For most teams in 2026, APIs win. The break-even point is higher than people expect. I looked for a published range of estimates and found none I could use.

Evidence caveat. No audited public break-even threshold for self-hosting against managed APIs holds up under source review. Published estimates disagree by about 50×, because each contains a different, unstated utilisation assumption. My reading is that break-even depends on utilisation more than on token volume. A GPU at 10% load carries about 10× the per-token cost of the same GPU at 100%. Re-run the numbers on your own traffic and target hardware before relying on any external figure.

October 2026 update: the open-weight gap widened on the newest measure. Through mid-2026 the gap narrowed: on Artificial Analysis's Intelligence Index v4 it was 3 points. Version 5, published 27 October with 60% of its evaluations on private test sets, puts it at 12 points, with Claude Opus 5.5 at 58 and the top open-weight model (Kimi K3.1 Pro) at 46. Kimi K3.1 Pro had not yet released its weights. Among models whose weights are out, DeepSeek V4.1 Flash 1017 and GLM-5.3 lead at 32, 26 points behind Opus 5.5 (Artificial Analysis, 27 Oct 2026). Gemini 4 Carbon, added the next day at 59.4, widens both gaps by 1.4 points (Artificial Analysis, 28 Oct 2026). Part of the widening comes from the switch to private test sets. Epoch AI's last general estimate is still a lag of 4 months, or 8 ECI points (May 2026). By 13 November the gap on the same index was 18 points: Claude Fable 5.2 (Max) at 64.6 against Kimi K3.1 Pro at 46.5 (Artificial Analysis data via LLM Frontier). Do not plan on a narrowing gap.

Agentic coding tells a different story. On Terminal-Bench 4.0, Kimi K3.1 Pro scores 63.64%. That is 1.5 points behind the leader, at about a third of the cost per solved task (see "What cost is"). Its list price is $2/$10 per million tokens, against $4/$20 for Claude Opus 5.5 (Moonshot). So the gap in cost per task comes as much from token use as from price. DeepSeek V4.1 Flash solves fewer tasks (31.82%) but costs about $1.70 per solve, at $0.15/$0.60 off-peak (DeepSeek).

Two cautions before you act on this. First, a hosted open-weight model is a managed API. A cheap hosted tier weakens the case for the frontier API, and it also weakens the case for running your own GPUs. Second, check the licence and weight status before planning a deployment. Moonshot describes Kimi K3.1 Pro as open-weight, but had not published a release date or licence by 31 October.

The direction of this lever is more reliable than any single number. Self-hosting can beat proprietary frontier APIs at lower volumes than it can beat cheap hosted open-weight APIs. Three things drive most of the variance: what you compare against, your sustained utilisation, and the fully loaded cost of running the stack.

The total cost of self-hosting is a multiple of the GPU price. Published breakdowns list the same hidden items:

Check any single number from a vendor blog against your own costs.

When self-hosting makes sense:

  1. Data residency requirements that no API can satisfy. HIPAA, financial regulators in specific jurisdictions, and GDPR Article 28 in some cases.
  2. Sustained high volume above the break-even for your comparison baseline. Verify with 30 days of real production traffic before procurement.
  3. Owned fine-tuned weights, where the IP value justifies the infrastructure. This is the best-documented path. Predibase's customer cases (Convirza, Checkr) and OpenPipe's customer pattern (per Kyle Corbitt, Sacra interview: Ramp, Notion, Databricks) belong here.
  4. Open weights meet the quality bar. In November 2026 the gap was about 18 points on Artificial Analysis's index (12 in October, and 26 in October for models with released weights) and about 1.5 points on agentic coding (see above). Verify on your own task.
  5. The latency floor matters. On-premise inference removes the round trip to a cloud provider.

Most production setups end up with a hybrid. Self-host the predictable bulk on tuned open weights. Use APIs for high-value, bursty or frontier-only work. Put one router between them.

Hybrid serving pattern: predictable bulk traffic on self-hosted open weights, high-value or frontier-only work on managed APIs.
Figure 6 The hybrid serving pattern. Predictable bulk traffic runs on self-hosted open weights. High-value or frontier-only work goes to managed APIs.

The serving stack matters as much as the model. If you self-host, three engines lead in 2026:

Avoid Ollama for production traffic. It is built for local development.

Quantisation is the cheapest serving optimisation. A 70-billion-parameter model such as Llama 3.3 70B needs about 140GB for its weights at 16-bit precision (two bytes per parameter). At 4-bit (AWQ or GPTQ) the weights take about 35GB, so the model fits on one 80GB GPU with room for the KV cache. Measure the quality drop on your own tasks.

Spot or preemptible GPU instances sell spare capacity at a discount, with the risk of interruption. They suit batch and evaluation work. Use checkpointing.


Lever 17. Speculative and parallel execution

C4.4. Do independent work at the same time.

Evidence: Peer-reviewed (Leviathan, Kalman and Matias, ICML 2023 oral: 2 to 3× acceleration at an identical output distribution; EAGLE, ICML 2024). Provider primary (vLLM and SGLang serving docs). Serving-engine benchmarks (vLLM EAGLE 3.1: 2.03× per-user output throughput; Snowflake Arctic Inference: 2.05 to 2.45× over non-speculative vLLM). No production case. See the caveat on multi-agent parallelism.

Run retrieval in parallel, run independent tool calls in parallel, and stream output to hide latency. These mostly improve latency. They also reduce user retries and abandonment, which cuts cost another way.

Evidence caveat on multi-agent parallelism. Anthropic's engineering post on its multi-agent research system tested an Opus orchestrator with parallel Sonnet sub-agents. It outperformed single-agent Opus by 90.2% on an internal evaluation. It also used about 15× more tokens than a chat conversation (Anthropic, 2025). This is parallel execution buying quality at a token premium. It belongs in this lever because the architecture is parallel. It is a quality case, with a cost premium. The cost-reduction half of this lever (speculative decoding) is supported by serving-engine benchmarks (vLLM, SGLang). I found no published production case study.

Anthropic's November results agree. With Claude Fable 5.2, a five-agent team reached a ProgramBench score of 0.6 in half the single agent's time, at a higher token cost. On DRACO, score followed total tokens across all three harnesses: teams scored higher because they used more tokens in the same time (system card §8.12, vendor-reported). Small teams buy speed and pay for it in tokens. Budget a multi-agent design as a cost increase until your own measurement shows otherwise.


C5. CALIBRATE

How do we know any of this works?

Lever 18. FinOps telemetry

C5.1. Attribute every dollar to a step, a tenant and an outcome.

Evidence: Peer-reviewed: none for cost attribution. Related work (Price Reversal Phenomenon, arXiv 2603.23971, preprint: list price mis-ranks true cost in 32% of model pairs; HAL, ICLR 2026: cost-aware evaluation). Industry primary (FinOps Foundation, State of FinOps 2026: 98% of 1,192 respondents now manage AI spend, up from 31% two years earlier). Production (Helicone, more than 2B interactions; Cloudflare per-user cost attribution).

This lever covers three things:

Log the model that served each request. The model you call and the model that answers can differ. With Claude Fable 5.2, a request blocked by a safeguard is answered by a fallback model (Opus 5 or Opus 4.8), and on the API the fallback is opt-in. In Anthropic's Terminal-Bench 4.0 run, 2.5% of requests fell back, which touched 10% of trials. In one external prompt-injection test, 53% of attempts did (system card §1.5, §8.5 and §5.2.1). Cost and quality attribution both need the serving model in the trace. So does routing (Lever 1).

The FinOps Foundation's 2026 State of FinOps report names AI and data platforms as the fastest-growing new category of enterprise spend. Token-based pricing brings volatility that legacy budgeting cannot handle.

What enterprises report. Gartner's survey of 2,002 CIOs, published 19 October 2026, found that only 13% report significant or transformational value from AI. Only 17% of their organisations have guidelines for abandoning low-value AI already in production (Gartner, 19 Oct 2026). A stopping rule needs cost per successful task for each use case. Without attribution there is nothing to stop on.

The macro picture. FinOps for AI is the top forward-looking priority for FinOps practitioners in the 2026 State of FinOps report (FinOps Foundation, Feb 2026). The Foundation changed its mission from "value of cloud" to "value of technology", to include AI and data spend. For the levers in this article, I read this as putting inference in cost of goods sold. Every cost lever becomes a margin lever as well as an efficiency one.


The evidence for each lever

Every one of the 18 levers is backed by published evidence. That means a public leaderboard, a peer-reviewed benchmark, a number the vendor disclosed, or a production case study. The strength of the evidence varies. Some levers have several leaderboards. Others have one production case I can cite. No lever depends on consultant opinion. The table at the end of this section names the strongest evidence for each one.

Five leaderboards are worth knowing by name. They anchor the cost and quality discussion for most of the framework.

Princeton HAL (Holistic Agent Leaderboard, ICLR 2026). This is the most relevant cost-aware leaderboard for agents. The ICLR paper covers nine benchmarks, and the live leaderboard lists more (SWE-bench, USACO, GAIA, τ-bench, ScienceAgentBench, Cybench, AgentHarm, CORE-bench and others). It plots accuracy against token cost as Pareto frontiers (Kapoor et al., arXiv 2510.11977). Each entry is tagged with whether it is on the frontier. I recommend this format for every internal evaluation. In my view, single-axis benchmarks are one of the main causes of cost overrun.

Status note. In late 2025 the Princeton team paused new-model updates to the main HAL leaderboard. They moved to the HAL Reliability Dashboard, which scores agents on consistency, predictability, robustness, safety and abstention. Their finding: "despite 18 months of model development, overall reliability shows only small improvements over time while accuracy steadily climbs." The Pareto format is still the right design. The live leaderboard is no longer the place to track new models.

Vals AI Terminal-Bench 4.0. This is the board behind the cost-per-solved-task table in "What cost is". It runs every model in the same minimal harness (mini-SWE-agent), averages three runs, and publishes cost per test next to accuracy (Vals AI). The harness is fixed, so cost differences come from the model and its token use.

Artificial Analysis Intelligence Index v5 (27 October 2026). It reports cost per task as a headline metric across nine evaluations. 60% of them use private test sets (Artificial Analysis). It also publishes each model at several effort levels, which shows the effort and cost trade-off in Lever 13. One finding stands out. Median output tokens per task rose from 35,000 to 59,000 between index versions. Token use per task can grow faster than prices fall.

Berkeley Function Calling Leaderboard (BFCL v4). This is the standard tool-use benchmark, and it now has cost and latency columns (Patil et al., gorilla.cs.berkeley.edu; OpenReview). The pattern matters for Lever 12 (tool design) and Lever 14 (fine-tuning). Databricks reports that smaller open-source models (DBRX, Llama-3-70B) can be lifted by careful prompting and parsing. In some aspects they come close to GPT-4 quality or pass it (Databricks, 2024).

Version note. BFCL v3 (multi-turn, late 2024) is now saturating, with Llama 3.1 405B at 88.5% and a tight cluster at the top. The small-model results above are on v3. BFCL v4 launched in April 2026 with "holistic agentic evaluation" and is much harder. Early v4 scores are far lower. When you cite BFCL, name the version. v3 numbers do not compare to v4 numbers.

This matches the harness evidence in Lever 11. Tool design, prompting and, where justified, fine-tuning set tool-use quality more than model size does.

NoLiMa and RULER (long-context benchmarks). These are the strongest support for Lever 5 (context engineering). Both measure how accuracy falls as context grows.

This is the evidence for context budgets. Filling context past the effective length costs more and also lowers quality. Lever 5 is one of the few cost levers where cost and quality improve together.

MTEB and MMTEB (Massive Text Embedding Benchmark). This is the public leaderboard for retrieval quality, and it bears directly on Lever 8 (MTEB leaderboard). The cost pattern as of April 2026:

So for most production RAG the useful question is "which is cheapest at the recall my evaluation requires". The answer is usually a smaller open-weight model.

Every lever, named evidence

I check each lever against three independent types of evidence.

A production case needs a deployment. Benchmark results from a named team count as benchmarks. A lever is Strong when all three are present. It is Adequate when one is missing or its mechanism differs from the claim. It is Thin when the anchors are workshop-grade or absent.

Lever Peer-reviewed Provider primary Production case Verdict
1. Model routing RouteLLM (ICLR 2025) AWS Bedrock IPR (63.6%) Uber GenAI Gateway; Cloudflare (10.57B tok/day) Strong
2. Deflection vCache (ICLR 2026, 12.5× hit rate at bounded error) Azure APIM semantic-cache policy; AWS ElastiCache (86% cost cut) Walmart waLLMartCache (ICPR 2024) Strong
3. Workflow vs agent HAL (ICLR 2026) Anthropic, "Building effective agents" None. Benchmark: smolagents (30% fewer LLM calls) Adequate
4. Confidence gating FrugalGPT (TMLR); UCCI (preprint, 31%) AWS Bedrock IPR None that matches. Cloudflare's risk tiers ($0.20 vs $1.68) use a rule-based classifier Adequate
5. Context engineering AgentDiet (FSE 2026); MemGPT Anthropic context-engineering guide Manus (100:1); OpenHands (about 50% cut) Strong
6. Retrieval engineering PLAID (CIKM 2022); BrowseComp-Plus Anthropic Contextual Retrieval (67%) None. Evaluation: Sourcegraph, long context against RAG Adequate
7. Caching PagedAttention (SOSP 2023); SGLang (NeurIPS 2024); Prompt Cache (MLSys 2024) Anthropic (90% / 85%) Notion, Simon Last; ProjectDiscovery (59%) Strong
8. Retrieval economics PLAID (CIKM 2022, 7×/45×) Qdrant Binary Quantization (32× memory, 40× faster) None. Leaderboards: MTEB / MMTEB, BEIR Adequate
9. Output shaping XGrammar (MLSys 2025) OpenAI Structured Outputs (100% vs <40%) None. Benchmark: Aider unified diffs (20% to 61%) Adequate
10. Prompt compression LLMLingua (EMNLP 2023); LongLLMLingua (ACL 2024) Microsoft Research (20×) None published Adequate
11. Harness engineering HAL (ICLR 2026, 21,730 rollouts) Anthropic harness-design guidance None. Vendor-run benchmarks: LangChain (+13.7 points); NVIDIA/LangChain (about 10× cheaper) Adequate
12. Tool design Toolformer (NeurIPS 2023); Gorilla (NeurIPS 2024) Anthropic tool-writing guide (65% token cut) None. Leaderboards: BFCL v4; τ2-bench Adequate
13. Reasoning discipline s1 (EMNLP 2025); HAL (ICLR 2026) Anthropic extended thinking / adaptive effort None. Leaderboard: Aider Polyglot (cost-tracked) Adequate
14. Fine-tuning / distillation QLoRA (NeurIPS 2023); LoRA (ICLR 2022) AWS Bedrock Distillation Predibase / Convirza (10×, vendor case study, no longer online); Together AI (2-3×) Strong
15. Service tiers Orca (OSDI 2022); PagedAttention (SOSP 2023) OpenAI Batch, Anthropic Batch (flat 50%) None. Arithmetic: Batch with caching reaches 2.5% of base Adequate
16. Self-hosted vs API DistServe (OSDI 2024); Orca (OSDI 2022) NVIDIA cost-per-token TCO None with a break-even figure; Uber (Ray + vLLM) shows the pattern Adequate
17. Speculative / parallel Leviathan et al. (ICML 2023); EAGLE (ICML 2024) vLLM / SGLang serving docs None. Serving-engine benchmarks: vLLM EAGLE 3.1 (2.03×); Snowflake Arctic (2.05-2.45×) Adequate
18. FinOps telemetry None for attribution. Related: Price Reversal (preprint); HAL (ICLR 2026) FinOps Foundation State of FinOps 2026 Helicone (2B+ calls); Cloudflare attribution Adequate
Evidence matrix: the eighteen levers checked against peer-reviewed, provider primary and production evidence. Five levers have all three; thirteen are missing one.
Figure 7 Each lever against the three evidence types. A filled dot means that type of evidence is present. Five levers have all three.

Five levers are Strong and thirteen are Adequate. None is Thin. Deflection (Lever 2) owes its main-track anchor to vCache, accepted at ICLR 2026. vCache also addresses the correctness problem that kept many teams from using semantic caching in production. It calibrates a threshold for each prompt, with an error bound the user sets.

Nine Adequate levers share one gap. Their third type of evidence is a benchmark, a leaderboard, an evaluation or a calculation, with no named deployment: Levers 3, 6, 8, 9, 11, 12, 13, 15 and 17. The other four fall short for their own reasons.

Benchmark lifecycle: leaderboards expire

By mid-2026, several of the most-cited leaderboards from 2024 and 2025 were saturated or compromised.

HumanEval, MBPP and MMLU are fully saturated. They no longer separate frontier models. The time from launch to saturation, or to pressure for a successor, is now about 12 to 18 months. Plan your evaluation infrastructure with that in mind. Four rules follow for any team building an internal evaluation.

  1. Pin every claim to a benchmark version. "Best on BFCL" is ambiguous; "best on BFCL v4" is precise. The same goes for Terminal-Bench 2.x against 4.0, and SWE-bench Verified against Pro.
  2. Note the harness. The same model in a different harness can score several points differently, and up to 12 across vendors on Terminal-Bench 4.0 (Lever 11). The leaderboard entry alone does not predict your production score.
  3. Treat contamination as the default risk. Benchmarks built from public issues (SWE-bench, Exercism-derived sets) appear in training data. For trends, prefer boards that refresh continuously (SWE-bench Live, SWE-Rebench, LiveCodeBench).
  4. Budget for benchmark migration. Internal evaluations built on SWE-bench Verified in 2024 now give a degraded signal. Assume you will swap the underlying benchmark every 12 to 18 months.

This is what happens when static datasets meet a field that improves this quickly. For cost work, my reading is this: a leaderboard lags model quality, and it signals where the field will publish a successor. Where you can, build internal evaluations on your own production traces.


Open problems

Six questions are still open across the 5 Cs. Each comes from the trade-offs above.

  1. Auto-budgeting for agent trajectories (C1 and C2). Today's best practice is a hand-tuned budget for each step. The open problem is a runtime that spreads a token budget across a trajectory as it runs, given a quality target and a cost cap. AgentDiet shows there is room (40 to 60% of tokens removable). The controller does not exist yet.

  2. Choosing the retrieval primitive at runtime (C2). Grep, vectors, reranking or direct corpus access is a fixed architecture choice today. The open problem is to make it a decision the harness takes for each query, based on the query and the corpus. Embedding-only retrievers fail on exact constraints. Pure grep fails on semantic queries. The answer is a router, and I have not seen one shipped.

  3. Harness portability across model upgrades (C3). Every harness encodes assumptions about what the model does reliably. As models improve, some of those assumptions become overhead. The production question is how to find the harness components a newer model no longer needs, and remove them, without rebuilding from scratch.

  4. Benchmark contamination as the default risk (C5). Three of the most-cited coding benchmarks of 2024 and 2025 were saturated or compromised within 18 months of launch. Internal evaluations built on public benchmarks degrade without warning. The open problem is a contamination-resistant method that a small team can run on its own production traces, without evaluation infrastructure the size of Scale AI's.

  5. Cost attribution by feature and tenant (C5). Cost per call is solved. Cost per successful task by feature, tenant or cohort is still open. The tooling has matured. Helicone reports more than 2B instrumented calls. Cloudflare documents per-user attribution through gateway metadata. OpenTelemetry graduated from the CNCF in May 2026 (CNCF, 21 May 2026), and its GenAI semantic conventions now cover LLM, agent and MCP spans. What is missing is a stable standard across systems. The GenAI conventions are still marked experimental. Attribution across a compound system with several providers is still fragile. A shared cache prefix, a hierarchical agent, or a tool that calls another agent each breaks naive attribution. In my view, the teams that solve this first will price AI features the way they now price cloud SKUs.

  6. Per-task cost prediction has a variance floor (C5). Token use varies between models on the same task, and between runs of one model. Most of the variation is in reasoning (arXiv 2603.23971). Adaptive effort adds another source, because the model now decides how much to think within the level you set. Part of this variance is noise in the process itself, so no predictor removes it. The open problem is budgeting, admission control and SLA design that treat per-query cost as a distribution with a long upper tail, and that degrade predictably on the expensive queries.

I think these are the questions worth research and engineering time in 2026.


Back to Part 2, Cost engineering for agentic AI, for the order of operations, the traps and what this means for enterprise programmes.

References

Routing & distillation research
Peer-reviewed anchors by lever
Realized Value series
Provider pricing and model documentation (checked 31 October 2026)
Cost-per-task benchmarks
Deflection and semantic caching
Open-weight versus closed-weight gap
Production cost cases
Benchmark contamination, saturation, and lifecycle
Function calling / tool use research
Trajectory reduction, agentic search, and 2026 retrieval research
Tokeniser and observability changes, 2026
FinOps macro reports, 2026

Change log

  • 19 Nov 2026 (v3, published). Lever 6 gains "The source data": the BEAVER benchmark on private company data warehouses (a preprint) and the HOH benchmark on outdated content (peer-reviewed), with a note that no study prices data preparation against the inference cost it saves. The verdict stays at Adequate. Two references added.
  • 18 Nov 2026 (v2, published). Plain-language review. 19 phrases reworded to remove figures of speech, for example "sits", "lands", "moves", "binds", "ships" and "survives" used for where a number is, how it changes, or what is left. The framework's own terms stay: ceiling, floor, binding constraint, discount, lever and bound-leak rate. Sentences shared with the main page were changed on both pages. No figures, claims or sources changed.
  • 15 Nov 2026 (v1, published). First published. The content moved here unchanged from Part 2 (v16), so that Part 2 reads in about 25 minutes. Figures are renumbered 1 to 7.
← Part 2 · Cost Part 3 · Reliability →