Chapter 11 — From Prototype to Production
TL;DR
- The gap between a working demo and a dependable production agent is vast — production means handling messy inputs, partial failures, policy changes, and real consequences repeatedly and reliably.
- Rollout strategies such as canary releases, shadow mode, and phased ring deployments let you control blast radius and earn autonomy through evidence, not optimism.
- Runtime safety controls — feature flags, approval gates (Preview then Commit), kill switches, and step/cost/time budgets — make probabilistic behavior manageable inside deterministic organisations.
- Observability across four layers (UX, scaffold, safety/evaluation, infra/cost) using correlated traces, dashboards by persona, and production replay turns a black-box agent into an auditable system.
- SLOs with error budgets, incident-to-regression workflows, and regular failure drills close the operational loop so every outage makes the agent stronger.
Agentic systems are easy to demo and hard to operate.
A demo proves the agent can do the task once. Production demands the agent can do the task repeatedly. It must handle messy inputs, changing policies, partial tool failures, real users, real audits, and real consequences.
Chapters 6–8 gave you the measurement and optimization stack: metrics, judges, datasets, gates, and improvement loops. Chapter 9 provided reusable design patterns. Chapter 10 gave you the UX contract: controls, evidence, and recovery. This chapter is the operational capstone. It shows how you wire those ideas into a dependable system that can ship, scale, and stay safe.
We will carry the same portfolio into production:
- HR Leave Agent (internal, policy-heavy)
- IT Service Desk Agent (event-triggered, exception-heavy)
- Invoice / Procurement Agent (money-touching, high risk)
- Contact Centre Case Assist (live, human-in-the-loop)
- CRM Co-pilot (drafts and updates, high-impact writes)
- Contract Review Agent (advisory, SME-heavy)
- Scheduling Agent (high volume, low to medium risk)
11.0 From demo to dependable system
11.0.1 Why productionizing agentic apps is different
A prototype lives in a clean room. It has curated policies, friendly prompts, stable tools, and happy-path inputs.
Production is the real company. HR Leave meets exceptions, side letters, union addenda, retroactive corrections, and colloquial questions. IT Service Desk meets noisy alerts, ambiguous tickets, missing fields, and brittle runbooks. Invoice meets OCR errors, novel vendor templates, partial POs, and workarounds that never made it into documentation. Contact Centre and CRM meet accents, interruptions, policy changes mid-call, and field-level CRM constraints. Contract Review meets bespoke clauses you have never seen. Scheduling meets "soft constraints" that exist only in people's habits.
You are putting probabilistic components inside deterministic organisations. Policies, SLAs, audits, and approvals expect repeatability and justification.
So you do not "productionize the model". You productionize the whole system:
model + scaffold + tools + runtime constraints + UX controls + evaluation gates + operations.
11.0.2 Production goals and non-negotiables
Before rollout, agree on two things per agent:
1) What "good" means in business terms
2) What must never happen in risk terms
Table 10.1 — Portfolio view: value, risk, and default autonomy
| Agent | Primary value | Typical failure cost | Non-negotiables | Default autonomy |
|---|---|---|---|---|
| HR Leave | Faster, consistent policy interpretation | fairness, policy breach | no statutory errors, no PII leakage | suggest or execute-with-approval |
| IT Service Desk | Faster triage and resolution | missed P1, wrong remediation | no unsafe automation on P1/P2 | constrained automation in low risk lanes |
| Invoice / Procurement | Throughput, fewer manual checks | financial loss, audit exposure | near-zero false approve for high risk tiers | advisory, constrained for small safe lanes |
| Contact Centre Assist | Better answers and summaries | bad advice, tone issues | groundedness, safe language, traceable sources | suggest-only, with strict evidence |
| CRM Co-pilot | Less admin work | incorrect high-impact write | Preview→Commit, audit logs, idempotency | execute-with-approval |
| Contract Review | Faster first pass | missed critical risk | high recall on red flags, provenance | suggest-only |
| Scheduling | Reduced coordination overhead | conflicts, annoyance | recoverability, transparency | constrained to low risk actions |
A simple operating principle that scales across domains:
No surprises: autonomy is visible, writes are previewed, failures are recoverable, and every decision has an evidence trail.
11.0.3 Production is a lifecycle, not a launch
Shipping an agent is a loop:
1) Evaluate offline using golden sets and adversarial suites 2) Ship with constraints: advisory mode, limited scope, approval gates 3) Observe in production through traces, overrides, and drift signals 4) Turn failures into regressions via incident-to-dataset workflows 5) Earn autonomy over time, and downgrade it when needed
This chapter shows how to build the mechanics that make that loop real.
11.0.4 Choosing Agentic AI pragmatically: cognitive complexity, integration complexity, and KPI readiness
Before you build, decide whether an agent is the right shape for the problem. Chapter 3, §3.1 covers the principles of problem selection; here we apply them as a production gate using three factors.
| Factor | Low | Medium | High |
|---|---|---|---|
| Cognitive complexity | Rules/forms, one tool, no ambiguity | Policy interpretation, 2–3 tools, clarifications | Many edge cases, 4+ tools, verification loops |
| Integration complexity | One system, clean APIs, limited writes | 2–3 systems, mixed data, gated writes | 4+ systems, unclear lineage, high-impact writes, strict audit |
| KPI readiness | Basic usage metrics | Adoption + unit economics | Full stack: adoption, unit economics, business/risk outcomes |
Higher cognitive complexity pushes you toward agentic UX patterns and evaluation gates. Higher integration complexity caps autonomy — do not grant high autonomy until tool contracts and governance are stable. If you cannot measure success, you cannot safely increase autonomy. Tie KPIs to your evaluation harness; ROI measurement should not be a separate spreadsheet.
Table 10.2 — Decision matrix: cognitive complexity × integration complexity
| Integration low | Integration high | |
|---|---|---|
| Cognitive low | Deterministic UX wins: forms, rules, simple retrieval. Agent optional for UX polish. | Workflow automation first: standardize data, fix schemas, build an integration gateway. Add an agent only after tool contracts stabilize. |
| Cognitive high | Copilot pattern: reasoning with evidence, suggest-only by default, limited tool surface. | Agentic system, but only with constraints: explicit knowledge layer, tool contracts, Preview→Commit, budgets, staged rollout, online eval, kill switches. |
Non-negotiable interpretation: if integration is high and tool contracts are unstable, autonomy stays low even if cognition is high.
Figure 10.1 — Decision flow for agentic AI adoption
flowchart TB
Q1["Is the task cognitively complex?<br/>Ambiguity, exceptions,<br/>multi-step reasoning"] -->|No| A1["Prefer deterministic UX<br/>or workflow automation"]
Q1 -->|Yes| Q2["Is integration complexity<br/>manageable? Stable tool contracts,<br/>clean data, clear ownership"]
Q2 -->|No| A2["Fix integration first:<br/>Tool schemas, knowledge layer,<br/>governance, data contracts"]
Q2 -->|Yes| Q3["Are KPIs and eval gates<br/>defined? ROI + safety<br/>+ drift monitoring"]
Q3 -->|No| A3["Define KPI stack and harness:<br/>Golden suites, online monitoring,<br/>rollback plan"]
Q3 -->|Yes| A4["Build with progressive autonomy:<br/>Suggest-only →<br/>execute-with-approval →<br/>limited auto"]HR Leave vignette: how the matrix drives a rollout plan
Target workflow: "Can I take childcare leave next Tue and Wed, do I need approval, and can you submit the request?"
The HR Leave agent has medium cognitive complexity due to policy interpretation, eligibility checks, approvals, and clarifications. It has medium-to-high integration complexity because it touches HRIS balances and submission, policy corpus, org chart, approval workflow, identity, and audit. KPI readiness must be explicit because statutory errors are unacceptable and ROI must be measurable.
Rollout plan implied by the matrix:
1) Phase 0: integration and data readiness - Tool contracts: HRIS read APIs, submission APIs with idempotency, approval workflow API - Knowledge layer: policy retrieval with citations and policy version selectors - Governance: RBAC, audit schema, retention and redaction rules
2) Phase 1: advisory pilot (dogfood) - Suggest-only: compute eligibility and balance, cite policy, provide next steps - KPIs: completion rate, HR escalation rate, override reasons, latency and cost budgets
3) Phase 2: execute-with-approval (Preview→Commit) - Draft leave request, show a structured preview and diff, require explicit approval - KPIs: correctness on statutory entitlement, tool failure rate, duplicate submission rate (must be zero)
4) Phase 3: limited automation lanes - Auto-submit only for low-risk leave types and cohorts, behind flags - Promotion criteria: stable offline evals plus green online signals, downgrade on drift or override spikes
This is how you avoid the common failure mode: pushing autonomy to compensate for missing integration maturity.
What to test before you scale autonomy
Use this as a pre-production checklist, then convert each item into a harness suite.
Cognitive complexity tests
- [ ] Ambiguous requests trigger minimal clarifying questions, not long dialogues
- [ ] Policy answers always cite the active policy version and clauses
- [ ] Deterministic calculators match HRIS truth for sampled cases
Integration complexity tests
- [ ] Tool schemas validate inputs and outputs with deterministic error codes
- [ ] Writes are idempotent and retry-safe
- [ ] Partial outages preserve state and provide a recovery path (retry, queue, handoff)
- [ ] Data quality checks catch missing IDs, stale balances, and mismatched leave codes
KPI readiness tests
- [ ] Baseline human time-to-complete is measured for the target workflows
- [ ] Human + AI time-to-complete is measured and sampled for quality
- [ ] Cost per successful case is tracked and bounded
- [ ] Business outcomes are tracked: compliance, rework, escalations, incidents
- [ ] Autonomy dial changes are correlated with KPI movement (and rollback is tested)
📚 Related sections: - Production challenges taxonomy → Appendix A - Failure case studies → Chapter 14, §14.6 - Evaluation foundations → Chapter 6
11.1 From harness results to rollout decisions
Your harness outputs are not just scores. They are rollout inputs.
11.1.1 Translate metrics into business and risk thresholds
The same number means different things in different workflows. For HR Leave, a small error rate on discretionary guidance might be tolerable if correction is easy. A small error rate on statutory entitlement is not. For Invoice, the trade-off is asymmetric: false approve is far worse than false escalate. For IT, misrouting P4 tickets is noise. Misrouting P1 is an incident.
Operationalize this with a threshold table per agent.
Table 10.3 — Threshold table with decision implications
| Metric | Target | Red line | Decision implication |
|---|---|---|---|
policy_violation |
0 | >0 | blocks promotion, forces downgrade |
pii_leakage |
0 | >0 | blocks promotion, immediate mitigation |
false_approve_high_risk |
~0 | >0 | advisory only until fixed |
override_rate |
decreasing | spike | tighten autonomy, investigate |
tool_failure_rate |
under threshold | spike | shift to fallback paths, open incident |
P95_latency |
within budget | breach | throttle, cache, downgrade model tier |
Two practical rules apply. Hard gates are "must be zero" or "must never regress". Soft gates are allowed to fluctuate, but must be trended and explained.
11.1.2 Cohorts are where surprises hide
Always cut results by cohorts that matter:
- role (manager vs employee)
- region and language
- department or business unit
- policy type and exception type
- vendor category (for invoice)
- ticket class and severity (for IT)
Most real production failures show up as "it works on average" but fails on one cohort that you did not isolate.
11.1.3 Promotion outcomes: no-go, go, go-with-guardrails
Most teams only think in "ship or don't ship". Agents need a third category: ship with constraints.
No-go means a hard gate fails or risk exposure is unacceptable. Go means the agent meets thresholds with margin, has low risk, and has clear recovery. Go with guardrails means ship, but reduce blast radius.
Guardrail patterns include:
- advisory-only mode
- limited scope (subset of queues, vendors, policy types)
- execute-with-approval only
- smaller tool surface
- lower step budgets and tighter timeouts
- conservative fallbacks
11.1.4 Rollout patterns that match the action class
A rollout is a product decision and an operations decision.
Table 10.4 — Rollout patterns and what they reveal
| Pattern | Best for | What you learn | Typical agents |
|---|---|---|---|
| Dogfooding | internal workflows | fast feedback, edge cases | HR, IT, invoice |
| Shadow mode | high-risk decisions | agreement and long-tail failures | invoice, IT routing |
| Ring deployment | staged adoption | cohort-specific issues | contact centre, CRM |
| Canary + fast rollback | risk-managed autonomy | drift, regressions | all, especially write paths |
| A/B or interleaving | UX impact | adoption, trust, productivity lift | contact centre, CRM |
📚 Related sections: - Scaffold versioning → Chapter 4, §4.1 - Evidence bundles → Chapter 12, §12.2 - Configuration registries → Chapter 12, §12.2.4
11.2 Versioning the whole system and shipping evidence
11.2.1 The unit of release is a configuration bundle
In production, "we upgraded the model" is not a release note. The real release unit includes:
- model and parameters
- scaffold prompts and routing logic
- tool schemas and allowlists
- runtime constraints (budgets, retries, sandbox)
- UX controls (Preview→Commit, approvals)
- policy corpora snapshots or policy version selectors
- evaluation config (datasets, rubrics, thresholds)
Treat this as a single versioned artifact: AgentBundle vX.Y.
11.2.2 Evidence bundles are how autonomy is earned
Every promotion should produce an evidence bundle that is easy to audit:
- scorecard deltas vs baseline
- failure set (top issues, "never again" cases)
- red-team results and mitigations
- tool safety tests, schema validity rates
- cost and latency projections
- rollout plan (ring sizes, flags, rollback plan)
If you cannot produce evidence, you do not increase autonomy.
📚 Related sections: - Online evaluation → Chapter 7, §7.3 - Autonomy UX → Chapter 10, §10.2 - Progressive autonomy → Chapter 3, §3.4
11.3 Control surfaces: kill switches, feature flags, autonomy dials, and budgets
Agentic systems need more control surfaces than traditional apps. Behavior can shift with data and context, not only code changes.
11.3.1 Control surfaces you need on day one
Four categories of control surfaces are essential:
1) Kill switches stop or downgrade unsafe behavior fast
2) Feature flags gate capabilities and scope
3) Autonomy dials make mode changes explicit and reversible
4) Budgets cap the probabilistic parts (steps, retries, context, time, cost)
Table 10.5 — Control surfaces and what they protect
| Control surface | What it protects | Example |
|---|---|---|
| Kill switch | immediate blast radius | disable CRM writes, keep drafting |
| Feature flag | controlled rollout | enable invoice auto-approve only for whitelisted vendors |
| Autonomy dial | predictable mode change | IT triage: label-only → auto-route low severity |
| Budgets | bounded execution | cap tool calls, cap retries, cap context size |
11.3.2 Kill switches must be tested, not documented
A kill switch is real only if it meets four criteria. It must be visible in an operator console. It must be permissioned so you know who can flip it. It must take effect quickly. The fallback behavior must be safe and usable.
Examples by agent type:
- Invoice: pause all auto-approve, route to manual queue
- HR: disable HRIS writes, keep advisory guidance
- Contact centre: disable outbound drafting, keep summarization
- IT: disable auto-close, keep triage suggestions
11.3.3 Autonomy dials are product features
Autonomy should not be a hidden config file. It should be a human-readable mode with explicit semantics.
Example for Scheduling:
- Suggest-only
- Auto-tentative within working hours
- Auto-confirm for known recurring patterns
Example for CRM Co-pilot:
- Draft-only
- Execute-with-approval (Preview→Commit)
- Automated for low-risk fields only
Tie every dial to gates. Increase autonomy only when offline evals are stable and online signals stay green. Downgrade autonomy on drift, incidents, or override spikes.
11.3.4 Budgets are production control surfaces
Budgets are how you make probabilistic behavior behave like software.
Table 10.6 — Budget types and typical responses when exceeded
| Budget | What it caps | Why it matters | Typical response |
|---|---|---|---|
| Step budget | total reasoning steps | prevents loops | escalate or stop with partial output |
| Tool-call budget | external actions | limits blast radius and cost | switch to advisory, ask for approval |
| Retry budget | transient failures | avoids runaway retries | fallback path, queue for later |
| Time budget | end-to-end latency | protects UX and SLA | degrade gracefully, show progress |
| Context budget | retrieved evidence | controls hallucination risk and cost | tighten retrieval, summarise evidence |
| Cost budget | token and tool spend | protects tenants and ops | route to smaller model tier |
Make budget breaches observable. A budget breach is an operational event, not an internal detail.
11.4 Observability and online evaluation: from traces to action
📋 Evaluation Pipeline: The HR Leave evaluation harness (Appendix E, §E.4) demonstrates patterns applicable to production monitoring: rule-based checks for fast validation, LLM judges for nuanced assessment, composite scoring with non-negotiable thresholds, and structured reporting for dashboards.
Observability is how you turn "a black box agent" into an auditable system.
11.4.1 A single run needs a standard envelope
For every unit of work (leave request, invoice, ticket, call, contract), log an envelope that connects:
- UX events
- scaffold steps
- tool calls
- model invocations
- policy checks and validators
- final outcome
Minimum correlation fields include run_id, case_id, trace_id, agent_bundle_version, autonomy_mode, tool_call_id per tool span, and evaluation_id when sampled into online eval.
11.4.2 Observe four layers, by design
1) UX layer: adoption, overrides, undo usage, feedback reasons
2) Scaffold layer: step paths, tool selection, retries, fallbacks
3) Safety and evaluation layer: guardrail triggers, sampled judge scores
4) Infra and cost layer: latency, error rates, token spend, tool saturation
If you only have layer 4, you know the system is slow. If you have layers 1–3, you know why quality changed.
11.4.3 Dashboards by persona
Different people operate different risks.
Table 10.7 — Dashboards by persona
| Persona | What they need | Example signals |
|---|---|---|
| Product owner | outcomes and adoption | success rate, override reasons, completion time |
| Ops / SRE | reliability | tool failure spikes, latency SLOs, queue growth |
| Risk / compliance | safety and policy | policy violations, PII events, unusual access |
| Domain SMEs | quality drift | sampled eval scores, new failure clusters |
11.4.4 Online evaluation closes the loop
Offline evals get you to pilot. Online evals keep you safe in production.
The common pattern works as follows. Sample production traffic into an evaluation queue. Run nearline judges or SME review with the same rubrics as offline. Trend scores, cluster failures, and link to replay tools.
Online signals should be wired to actions:
- tighten autonomy
- flip flags
- activate kill switches
- roll back an AgentBundle version
- expand SME review for affected cohorts
11.4.5 Replay is for engineering, forensics, and governance
When something goes wrong, you need two answers. First, what happened exactly. Second, would it still happen now.
Replay lets you rerun a trace against the same bundle to reproduce and explain. You can also rerun against the candidate bundle for regression and fix validation. Your "forensics corpus" becomes your highest value regression suite.
📚 Related sections: - Trace structure → Chapter 5, §5.3 - Online evaluation → Chapter 7, §7.3 - Audit requirements → Chapter 12, §12.2
11.5 Reliability and recovery: UX recovery plus backend guarantees
Chapter 10 introduced recovery as a UX requirement. Production demands recovery as an end-to-end system property.
11.5.1 Preview→Commit is the default for high-impact writes
High-impact writes (CRM updates, invoice approvals, HRIS submissions) should follow a contract:
1) Agent generates a preview with a structured diff 2) Runtime validates policy and permissions 3) User approves (or a workflow orchestrator approves) 4) Runtime commits with idempotency 5) Audit log is written regardless of outcome
This is where UX pattern, tool contract, and runtime enforcement must align.
11.5.2 Idempotency, compensation, and state preservation
Recovery fails when the system cannot safely retry.
Idempotency keys prevent duplicate side effects on retry. Compensating actions allow an undo path when downstream supports it. Preserved state ensures the user does not restart from scratch after a tool timeout.
A practical goal:
Every failure type renders a recovery path with preserved state and a safe next action.
11.5.3 Failure drills and game days
Run drills per agent:
- simulate a tool outage (HRIS, ERP, CRM)
- simulate drift (policy change, new vendor template)
- simulate unsafe behavior (prompt injection, tool misuse attempt)
Drills must exercise four capabilities. Detection uses dashboards and alerts. Response uses flags, kill switches, and autonomy downgrade. Investigation uses replay and trace inspection. Prevention converts failures into new regression cases and updated rubrics.
📚 Related sections: - Model fallback patterns → Chapter 9, §9.6B - Recovery UX → Chapter 10, §10.5 - Tool resilience → Chapter 9, §9.2
11.6 Cost and Latency Engineering
Cost is not just a finance concern. It is a reliability constraint and a UX constraint.
An agent that costs too much per task will be bypassed, throttled, or killed during peak load. An agent that takes too long will lose user trust. Cost and latency engineering is how you make agentic AI sustainable at scale.
11.6.1 The cost anatomy of an agent run
Every agent run has a cost structure you can decompose and optimize.
Table 10.8 — Cost components and optimization levers
| Cost component | What drives it | Optimization levers |
|---|---|---|
| Input tokens | Context size: system prompt, retrieved docs, conversation history, tool schemas | Context budgets, smarter retrieval, compaction |
| Output tokens | Response length, reasoning verbosity, structured output overhead | Output caps, schema constraints, step limits |
| Reasoning tokens | Chain-of-thought, planning, self-critique (if using reasoning models) | Model selection, when to reason vs when to retrieve |
| Tool calls | Number of external API calls, retries, polling | Tool batching, caching, fail-fast patterns |
| Embedding/retrieval | Queries per step, corpus size, reranking passes | Query consolidation, tiered retrieval, embedding caching |
| Compute/hosting | Model size, GPU time, orchestration overhead | Model routing, batching, serverless vs reserved |
HR Leave example cost breakdown:
- Intake step: ~500 input tokens (user message + minimal context)
- Policy retrieval: ~2,000 input tokens (retrieved clauses) + embedding query cost
- Eligibility check: ~3,000 input tokens (context + balance data) + ~500 output tokens
- Draft response: ~1,500 input tokens + ~800 output tokens
- Total per request: ~7,000 input + ~1,300 output + 2–3 tool calls + 1–2 retrieval queries
Track this per-step breakdown, not just aggregate cost per conversation.
11.6.2 Model routing: right model for the job
Not every step needs your most capable model. Chapter 9, §9.6A covers model routing patterns in detail — selection by task complexity, fallback cascades, and graceful degradation. Here we summarize the production-specific considerations.
Table 10.9 — Model routing taxonomy
| Step type | Model tier | Rationale |
|---|---|---|
| Classification/routing, extraction | Small, fast | High volume, low ambiguity |
| Query formulation, retrieval | Small to medium | Bounded creativity, low risk |
| Reasoning/judgment, explanation | Medium to large | Nuance required, quality matters for trust |
| Planning, critique/verification | Large | Broad context, catches errors before commit |
Escalation triggers: confidence below threshold, retry after failure, or high-stakes cohort (default to larger model). Testing: A/B test model tiers per step with quality held constant; track escalation rate as a routing accuracy indicator; regression suite must pass on the smallest acceptable model.
11.6.3 Caching strategies
Caching is the highest-leverage cost optimization. But it only works if invalidation is correct.
Table 10.10 — Cache types and invalidation guidance
| Cache type | What it stores | Invalidation trigger | TTL guidance |
|---|---|---|---|
| Prompt cache | Common system prompts, tool schemas | Prompt version change | Days to weeks |
| Semantic cache | Query → response for common questions | Policy update, corpus change | Hours to days |
| Embedding cache | Document chunk → embedding vector | Document update, re-chunking | Until source changes |
| Tool result cache | Stable lookups (e.g., holiday calendar, org chart) | Source system update | Hours to days |
| Session cache | Parsed user context, extracted fields | Session end | Session duration |
Prompt caching (provider-level):
Most model providers now offer prompt caching for repeated prefixes. OpenAI caches automatically for identical prefixes. Anthropic uses explicit cache_control markers. Cost savings are typically 50–90% on cached input tokens.
Structure your prompts to maximize cache hits:
[SYSTEM PROMPT — stable, cacheable]
[TOOL SCHEMAS — stable, cacheable]
[POLICY EXCERPTS — semi-stable, per-policy-version]
[USER CONTEXT — dynamic, not cached]
[CONVERSATION — dynamic, not cached]
Semantic caching for common queries:
For high-volume agents, many queries are near-duplicates. Examples include "How much annual leave do I have?" phrased 50 different ways, or "Can I take leave next Friday?"
The pattern works as follows:
- Embed incoming query
- Check similarity against cache keys (threshold ~0.95)
- If hit and cache fresh: return cached response
- If miss or stale: generate, cache result with query embedding as key
Invalidation discipline:
Every cache entry needs an invalidation contract. When a policy document is updated, invalidate all policy-related semantic cache. When a tool schema changes, invalidate prompt cache. When a user's leave balance changes, invalidate that user's balance lookups.
Track cache hit rate and "stale serve rate" as operational metrics.
11.6.4 Context budget management
Context is your most expensive input. Manage it like a budget, not an open bar.
Budget allocation per step:
# Example: HR Leave Agent context budgets
steps:
intake:
max_context_tokens: 2000
includes: [system_prompt, user_message, minimal_history]
policy_retrieval:
max_context_tokens: 4000
includes: [system_prompt, query, top_k_chunks]
top_k: 5
eligibility:
max_context_tokens: 6000
includes: [system_prompt, request_summary, policy_excerpts, balance_data]
response:
max_context_tokens: 3000
includes: [system_prompt, decision, evidence_summary, response_guidelines]
Compaction strategies:
When context exceeds budget:
- Summarize history: Replace older conversation turns with a summary
- Prune retrieval: Reduce top_k or filter by relevance threshold
- Extract decisions: Replace verbose reasoning with structured decisions
- Pointer references: Store full content in memory, pass only IDs and summaries
HR Leave example compaction:
# Before compaction (verbose)
"User asked about leave balance on Jan 5. I retrieved policy section 3.2 which states...
[500 words of policy]. The user clarified they want childcare leave. I then..."
# After compaction (structured)
{
"decisions": ["leave_type: childcare", "dates: Jan 15-16"],
"evidence_refs": ["policy_v2.3_section_3.2", "balance_check_2024-01-05"],
"open_questions": [],
"next_step": "assess_eligibility"
}
11.6.5 Latency engineering
Latency has two components: model inference time and everything else.
Table 10.11 — Latency budget by component
| Component | Target | Typical range |
|---|---|---|
| User perception threshold | <3s for first response | — |
| Intake + routing | <500ms | 200–800ms |
| Retrieval (per query) | <300ms | 100–500ms |
| Model inference (small) | <1s | 500ms–2s |
| Model inference (large) | <3s | 1–5s |
| Tool call (external API) | <2s | 500ms–5s |
| End-to-end (simple query) | <5s | 3–8s |
| End-to-end (complex multi-step) | <15s | 10–30s |
Latency reduction patterns:
- Streaming: Start showing output before generation completes
- Parallel retrieval: Fire multiple retrieval queries concurrently
- Speculative execution: Start likely next steps while awaiting confirmation
- Warm pools: Keep model connections warm, avoid cold start
- Edge caching: Cache common responses close to users
- Async tool calls: For non-blocking tools, fire and continue
Streaming for perceived latency:
# Stream partial responses as they generate
async def stream_response(agent, query):
yield {"type": "status", "message": "Checking your leave balance..."}
balance = await agent.get_balance()
yield {"type": "status", "message": "Reviewing policy..."}
async for chunk in agent.generate_response_stream():
yield {"type": "content", "chunk": chunk}
11.6.6 Cost-per-successful-task as the north star
Raw token cost is a vanity metric. What matters is cost per successful task completion.
Cost efficiency = Total cost / Successful completions
Where:
- Total cost = tokens + tool calls + infra + human escalation handling
- Successful = task completed correctly without human intervention or rework
Track this metric by cohort (leave type, employee grade, complexity tier), by time period (daily, weekly trending), and by model configuration (to compare routing strategies).
Cost–quality frontier:
Maintain a frontier chart showing cost vs quality trade-offs:
Quality (task success %)
|
100%| * Current (expensive)
| *
95%| * * Target (efficient)
| *
90%| * (too cheap, quality suffers)
|________________
$0.10 $0.50 $1.00 Cost per task
Every cost optimization should be validated against this frontier. You want to move left (cheaper) without moving down (worse quality).
11.6.7 Rate limits and back-pressure
When the agent is saturated, define graceful degradation.
Table 10.12 — Saturation levels and responses
| Saturation level | Response |
|---|---|
| Normal | Full agent capability |
| Elevated (>80% capacity) | Disable non-essential features, simplify retrieval |
| High (>95% capacity) | Queue new requests, show wait time estimate |
| Critical | Route to smaller model, advisory-only mode |
| Overload | Fail fast with clear message, offer manual fallback |
Implementation guidance: use token bucket or leaky bucket rate limiters. Expose queue depth and wait time in UX. Have a "degraded mode" flag that simplifies agent behavior. Never silently drop requests.
11.6.8 Cost observability
Add cost tracking to your telemetry:
# Per-run cost telemetry
run_cost:
run_id: "abc123"
input_tokens: 7234
output_tokens: 1456
reasoning_tokens: 0
model_costs:
- model: "gpt-4o-mini"
calls: 3
cost_usd: 0.002
- model: "gpt-4o"
calls: 2
cost_usd: 0.045
tool_costs:
- tool: "hris_lookup"
calls: 2
cost_usd: 0.001
retrieval_costs:
queries: 2
cost_usd: 0.0005
total_cost_usd: 0.0485
outcome: "success"
cost_per_success_usd: 0.0485
Table 10.13 — Cost dashboards by persona
| Persona | Cost view |
|---|---|
| Product owner | Cost per task by use case, trending, budget burn rate |
| Engineering | Cost by model, by step, by cache hit/miss |
| Finance | Total spend, forecast, cost vs budget |
| Ops | Cost spikes correlated with incidents, anomaly detection |
📚 Related sections: - Model routing patterns → Chapter 9, §9.6A - Context budgets → Chapter 9, §9.4B - Runaway cost failure → Chapter 14, §14.6.1
11.7 Production readiness gate: the capstone checklist
A production gate should not be "we feel confident". It should be a concrete checklist tied to artifacts you already introduced in Chapter 9 and to gates you defined in Chapters 6–7.
11.7.1 Readiness checklist (minimum viable production)
Specification and contracts
- Agent charter (scope, non-goals, action classes)
- ToolSpecs with schemas, allowlists, idempotency for writes
- Memory and context policy (what is stored, TTL, redaction)
Evaluation and evidence
- Golden set and risk suites exist
- Hard gates defined (policy, PII, unsafe actions)
- Evidence bundle produced for the release
Controls
- Kill switches and autonomy dials exist and are permissioned
- Feature flags for new capabilities and cohorts
- Budgets configured (steps, retries, time, context, cost)
Observability
- Run envelope and correlation IDs implemented
- Dashboards exist for product, ops, risk personas
- Replay tooling exists for at least the critical workflows
Recovery and operations
- Preview→Commit for high-impact writes
- Runbooks exist for top failure modes
- At least one failure drill completed
If any category is missing, ship in a more constrained mode, or do not ship.
📚 Related sections: - Governance checkpoints → Chapter 12, §12.2 - Evaluation gates → Chapter 7, §7.4 - Evidence bundles → Chapter 12, §12.2.2
11.8 Bridge to governance and whole-system lifecycle (Chapter 12)
By the end of this chapter, your agent portfolio should be:
- measurable through harnesses and online eval
- controllable via flags, dials, budgets, and kill switches
- observable with traces, dashboards, and replay
- recoverable using Preview→Commit, idempotency, and compensation
- operable through runbooks, drills, and change management
Chapter 12 zooms out to governance checkpoints, lifecycle management, and how these artifacts become assurance evidence for auditors, regulators, and internal risk owners.
In other words: Chapter 11 builds the operating machine. Chapter 12 defines how the organisation runs it, safely and continuously.
10.A HR Leave: Production patterns in one end-to-end walkthrough
This section makes the production patterns in Chapter 11 concrete using a single HR Leave request. It is written like a runbook with numbered callouts, non-negotiables, and a checklist you can turn into harness tests.
📚 Pattern foundations: This section assumes familiarity with the HR Leave artifact set and pattern stack. For the underlying recipe—including prompt specs, tool specs, skill registry, and harness structure—see Chapter 9, §9.9. This section builds on that foundation with production-grade operational detail.
Scenario: An employee asks in Teams, "My child is sick. Can I take childcare leave next Tue and Wed? Do I need manager approval?"
In a demo, the agent answers from a static policy excerpt. In production, the agent must reliably do all of this under messy inputs, partial outages, and shifting policy versions, while preserving trust.
10.A.1 End-to-end walkthrough (one request, many production realities)
Step 0: Establish scope and mode
The UI displays the current autonomy mode. The default is suggest-only or execute-with-approval. The agent declares what it will do next: "I will check your eligibility and balance. If you want, I can draft the leave request for approval."
Step 1: Intake and intent parse (bounded)
Extract leave type candidates (childcare leave vs annual leave vs unpaid), dates, urgency, and any missing required fields. If required fields are missing, ask a single targeted clarification question. Do not search or call tools until you know what you need to look up.
Step 2: Build step-specific context (the knowledge layer)
Production HR Leave needs two kinds of context. The semantic layer (structured) includes employee profile (country, grade), entitlement rules, leave balance, and approval rules from HRIS. Enterprise search (unstructured) includes HR policy docs, side letters, and FAQs from SharePoint or Confluence, retrieved with citations.
Step 3: Eligibility + entitlement compute (deterministic where possible)
Use deterministic calculators for remaining leave balance, eligibility rules based on profile and policy version, and approval requirement based on leave type and dates. The model is used for interpretation and explanation, not arithmetic and policy logic.
Step 4: Draft the outcome with provenance
The agent produces a short answer ("Yes, you can take childcare leave for Tue/Wed; manager approval is required."). It also provides a structured rationale linked to policy clauses with citations, a confidence indicator (high/medium/low) based on retrieval and rule coverage, and next actions ("Draft leave request for review" or "Escalate to HR due to exception").
Step 5: Preview → Commit for HRIS writes
If the user requests submission, generate a preview of the HRIS write. This includes leave type, dates, reason, approver, and supporting doc requirement. Show a diff-like view of what will be created or changed. Run validations for policy constraints, permission checks, and idempotency key creation so retries cannot double-submit.
Step 6: Approval gate
In execute-with-approval mode, the user confirms the preview. If required, the manager approval workflow is triggered with status pending.
Step 7: Commit + audit
Commit to HRIS with idempotency and safe retry semantics. Write an audit record that includes agent bundle version, autonomy mode, tool calls executed, evidence references (policy version, citation IDs), and approvals captured.
Step 8: Exception recovery path (preserve state)
If HRIS is down or a tool call fails, the agent does not restart. It renders a recovery UI with preserved state offering three options: retry now, submit later (queue), or handoff to HR with context pack. The draft request remains intact and traceable.
10.A.2 Numbered callouts (production patterns in this walkthrough)
These ten callouts expand on the six-step pattern plan in Chapter 9, §9.9.3. Where Chapter 9 focuses on which patterns to apply, these callouts focus on why they matter in production and what goes wrong without them.
1) Knowledge layer is a product. Combine structured HRIS semantics with unstructured policy search with citations. Treat retrieval quality and policy versioning as release artifacts, not "data plumbing".
Note: Chapter 9's "Context assembly" step (§9.9.3, Step 2) covers the pattern mechanics. This callout elevates knowledge layer to a first-class product concern—if retrieval quality degrades, the entire agent fails, regardless of how good the model is.
2) Compounding reliability is why runs must be short. Multi-step workflows fail quickly when each step is imperfect. Use checkpoints: validate early, ask one clarification, and gate writes.
3) Single orchestrator, modular tools. One planner orchestrates retrieval, entitlement calculator, approval rules, and HRIS API. Avoid premature multi-agent decomposition unless evals show net gain.
4) State machine tracks provenance and confidence. Store what evidence was used and how certain the agent is. Define rollback points: before approval, before commit, after commit.
5) Observability must include decision traceability. Log not only "what happened" but "why" through the decision chain, evidence set, and confidence.
6) Contextual integrity: should-use checks beyond can-access. "Can I see teammate leave balance?" must be refused even if the system can query it. Enforce intent-based data use rules outside the prompt.
7) Token economics is a UX constraint. Budgets prevent runaway retrieval and long reasoning loops. Route simple extraction and classification to smaller models. Use larger models for explanations.
8) Progressive autonomy. Start advisory, then execute-with-approval, then limited auto-submit only for low-risk lanes. Autonomy upgrades must be evidence-based and reversible.
9) SME correction loop. HR policy owners can flag wrong answers and attach the right clause. Convert incidents into regression cases automatically.
10) Second agent should be easier than the first. HR Leave becomes the template: tool contracts, auth, telemetry, release bundle, and dashboards. Production capability is a platform, not a one-off.
10.A.3 Non-negotiables (HR Leave agent)
These are the ship gates for production HR Leave.
1) No unverifiable policy claims. All policy-based answers must include citations to the active policy version.
2) No high-impact writes without Preview → Commit. All HRIS writes require a preview, validation, and explicit approval.
3) No hidden autonomy. Autonomy mode must be visible and stored with the run.
4) No policy enforcement by prompt alone. Deterministic runtime guards block forbidden actions regardless of model output.
5) No loss of user progress on failure. Tool failures must yield a recovery UI with preserved state.
6) No double submissions. HRIS commits must be idempotent and safe to retry.
7) No cross-person data leakage. Enforce contextual integrity for HR data access.
8) Every run is replayable. The run envelope must allow reconstruction of the decision chain, evidence set, tool calls, and approvals.
10.A.4 What to test (checklist you can turn into harness suites)
Use this as an acceptance test list and as a basis for your regression suites.
A. Knowledge + retrieval
- [ ] Correct policy version is selected for employee cohort (country, grade, union addendum)
- [ ] Citations are always present for policy claims
- [ ] Retrieval returns relevant clauses for common queries and edge cases
- [ ] Evidence set is bounded (context budget respected)
B. Eligibility + entitlement correctness
- [ ] Entitlement calculator matches HRIS truth for sampled cases
- [ ] Leave type classification is correct for ambiguous phrasing
- [ ] Approval requirement is correct across cohorts and blackout periods
C. Write safety (Preview → Commit)
- [ ] Preview shows a diff-like summary of the HRIS write
- [ ] Commit is blocked if validation fails (missing fields, policy violation, auth)
- [ ] Idempotency prevents duplicate submissions on retry
- [ ] Audit log entries exist for preview, approval, and commit
D. Autonomy + UX transparency
- [ ] Autonomy mode is always visible in UI and recorded in telemetry
- [ ] Suggest-only mode never calls HRIS write tools
- [ ] Execute-with-approval mode always requires explicit approval
E. Recovery + resilience
- [ ] Tool outage renders recovery UI (retry/queue/handoff) with preserved draft
- [ ] Partial failures do not corrupt state or create duplicate requests
- [ ] Retry budget caps are enforced and observable
F. Security + contextual integrity
- [ ] Requests for other employees' leave balances are refused, even if data is accessible
- [ ] Only least-privilege HRIS scopes are used for tool calls
- [ ] Prompt injection attempts in policy documents do not change tool behavior
G. Observability + replay
- [ ] Each run has run_id, trace_id, case_id, agent bundle version, autonomy mode
- [ ] Decision chain is recorded: leave type chosen, evidence IDs, confidence
- [ ] Tool calls are traceable with redaction and deterministic error taxonomy
- [ ] Replay reproduces the same decision points under the same bundle and data snapshot
10.A.5 Visual: HR Leave production flow with checkpoints and recovery
Figure 10.2 — HR Leave production flow
flowchart TB
U["Employee asks in Teams"] --> I["Intake + intent parse"]
I --> K["Knowledge layer: HRIS semantics + policy search with citations"]
K --> E["Eligibility + entitlement compute"]
E --> P["Draft outcome + provenance + confidence"]
P -->|"Suggest-only"| R["Answer + next steps"]
P -->|"Execute-with-approval"| V["Preview HRIS write + validations"]
V --> A["Approval gate"]
A --> C["Commit to HRIS (idempotent)"]
C --> L["Audit log + run envelope + traces"]
L --> R
C -. "HRIS outage" .-> X["Recovery UI: preserve draft<br/>Retry | queue | handoff"]
X --> VThe knowledge layer is explicit, not implicit. The write path is gated by Preview → Commit and approval. Failures route to recovery without losing state. Observability is attached to the commit path and the response path.
📚 Related sections: - Complete HR Leave reference → Appendix B - Scaffold design → Chapter 4, §4.3 - Best practices applied → Chapter 9
Reference map
📚 Full reference map: The detailed source reference map for Chapter 11 (covering 25 topics with primary sources) has been consolidated in Appendix F, §F.3.
Visual: Production control surfaces for an enterprise agent
Figure 10.3 — Production control surfaces
flowchart TB
A[AgentBundle: prompts + toolspecs + policies + eval gates + configs] --> B[Staged rollout]
B --> B1[Canary / rings]
B --> B2[Shadow traffic]
B --> C[Runtime controls]
C --> C1[Feature flags + autonomy dial]
C --> C2[Approval gates + Preview→Commit]
C --> C3[Budgets: steps, retries, timeouts]
C --> D[Observability envelope]
D --> D1[Traces: steps + tool calls]
D --> D2[Metrics: latency, cost, error rates]
D --> D3[Logs: prompts, responses, tool I/O]
D --> E[Ops loop]
E --> E1[SLOs + error budgets]
E --> E2[Incident response + postmortems]
E --> E3[Replay + regression suites]
E --> F[Evidence bundle for governance]Key Takeaways
- Productionize the system, not the model — The unit of release is the entire AgentBundle (model, prompts, tool schemas, runtime constraints, evaluation config, and UX controls), versioned and shipped together.
- Earn autonomy through evidence — Every promotion from advisory to execute-with-approval to limited automation must be backed by an evidence bundle of offline evals, red-team results, and online signals, and must be reversible on drift.
- Control blast radius with layered runtime surfaces — Kill switches stop unsafe behavior immediately, feature flags gate scope, autonomy dials make mode changes explicit, and budgets cap the probabilistic parts of execution.
- Observe four layers, not just infrastructure — UX events, scaffold decisions, safety/evaluation signals, and infra/cost metrics together explain not only that something broke but why quality changed.
- Make cost and latency first-class engineering concerns — Track cost-per-successful-task (not raw token spend), route steps to right-sized models, cache aggressively with correct invalidation, and define graceful degradation under saturation.
- Close the operational loop with incidents and drills — Wire SLOs and error budgets to concrete actions (tighten autonomy, flip flags, roll back bundles), run failure drills per agent, and convert every postmortem into new regression cases and updated rubrics.
- The second agent should be easier than the first — Treat production tooling (tool contracts, auth, telemetry, release bundles, dashboards, and replay) as a reusable platform so each new agent inherits operational maturity rather than rebuilding it.
Next → Chapter 12 addresses governance, policy, and lifecycle — ensuring agents ship on the right terms and remain safe over time.