Appendix F — Evaluation Quick-Reference Templates and Source Map
Version: 1.0 Last updated: February 2026
Navigation: ← Appendix E | Appendix Index | Appendix G →
This appendix contains the quick-reference templates and curated source map from Chapter 7 (Evaluation in Practice). They are extracted here to keep Chapter 7 focused on concepts and patterns while providing copy-paste reference material in one place.
F.1 Quick Reference Templates
These templates are copy-paste starting points. Adapt column definitions, thresholds, and owners to your domain.
Table F-1. Metrics registry template
Use this template to define your metrics registry. Each row defines one metric with its evaluator, owner, and gate threshold.
| Metric name | Level | Definition (pass/fail preferred) | Evaluator type | SME owner | Data source | Gate threshold | Notes |
|---|---|---|---|---|---|---|---|
policy_correctness |
Task | Output complies with leave policy for the scenario | LLM-as-judge + rubric | HR policy SME | Golden cases + sampled traces | Hard: ≥ 0.95 | Add counterexamples for edge rules |
citation_faithfulness |
Task | Claims supported by retrieved excerpts | LLM judge + citation checker | Knowledge owner | RAG traces | Hard: ≥ 0.90 | Penalise invented citations |
tool_safety |
Step | Tool calls are least-privilege, correct target, correct parameters | Rules + allowlist | Platform security | Tool spans | Hard: 0 critical | Alert on privilege escalation |
handoff_quality |
Trace | When escalated, handoff includes required context fields | Rule check | Ops lead | Ticket spans | Soft: ≥ 0.85 | Drives training and UX |
Table F-2. Gate policy matrix template
Use this template to define what runs at each stage and what gates apply.
| Stage | What runs | Hard gates (block promotion) | Soft gates (warn, triage) | Owner |
|---|---|---|---|---|
| PR | Unit checks + small regression set | Safety violations, PII leak, schema invalid | Style, verbosity, minor UX rubrics | Eng lead |
| Nightly | Full regression + journey eval | Critical workflow failures, tool misuse | Known issues trend, minor rubric shifts | Eval engineer |
| Pre-release | Red-team + expanded regression | Jailbreak success, privilege escalation, critical policy errors | Non-critical "near misses" | Product + security |
| Post-release | Shadow sampling + canary | Metric lower bound crosses threshold, spike in guardrail triggers | Drift signals, new error clusters | On-call + product |
Table F-3. Synthetic data worksheet (dimensions approach)
Use this template when you have limited production data or when you want systematic coverage. Define dimensions, then generate tuples by cross-product and filtering, or by direct tuple generation.
| Dimension | Example values | Why it matters | HR Leave example |
|---|---|---|---|
| User intent | policy lookup, eligibility, calculation, exception | Different reasoning and retrieval needs | "Can I carry forward leave?" vs "How many days left?" |
| Employment type | full-time, part-time, contract | Policy branches | contract staff probation rules |
| Leave type | annual, medical, childcare, unpaid | Constraints differ | childcare requires age limits |
| Time boundary | past, current, future | Temporal reasoning | "in next FY" carryover |
| Evidence availability | answerable, unanswerable | Calibration, abstention | policy not in KB for subsidiary |
| Risk level | low, medium, high | Gate severity | medical leave, PII |
F.2 Chapter 7 Reference Map
This table maps every major concept in Chapter 7 to its engineering angle and primary sources. Use it to find authoritative references for any evaluation topic covered in the chapter.
F.3 Chapter 8 Reference Map (Optimization)
This table maps every major concept in Chapter 8 to its engineering angle and primary sources. Use it to find authoritative references for any optimization topic covered in the chapter.
| Concept | Primary sources (clickable) |
|---|---|
| 8.1 The optimization landscape | |
| What can be optimized (prompts, tools, scaffolds, skills, configs) | Anthropic, Demystifying evals for AI agents | OpenAI Cookbook, Optimize prompts |
| Manual vs automated optimization | OpenAI, Prompt optimizer | DSPy Docs, Optimization |
| Three modes (single-task, multi-task, generalization) | GEPA paper, arXiv:2507.19457 | AlphaEvolve, Google DeepMind |
| 8.2 Actionable Side Information (ASI) | |
| Beyond scalar rewards | GEPA paper, arXiv:2507.19457 |
| Designing rich evaluator feedback (traces, intermediates, diagnostics) | OpenAI, Trace grading | Hamel Husain and Shreya Shankar, LLM Evals FAQ |
| ASI as gradient equivalent for text optimization | GEPA paper, arXiv:2507.19457 |
| 8.3 Optimization algorithms | |
| DSPy MIPROv2 (program-level optimization) | DSPy Docs, Optimization | DSPy paper, arXiv:2310.03714 |
| GEPA (reflective prompt evolution) | GEPA paper, arXiv:2507.19457 | gepa-ai.github.io/gepa |
| AlphaEvolve (LLM-evolution for algorithms) | AlphaEvolve, Google DeepMind |
| OpenEvolve / ShinkaEvolve (open-source evolution) | OpenEvolve (GitHub) | ShinkaEvolve (GitHub) |
| optimize_anything unified API | gepa-ai.github.io/gepa |
| 8.4 Trajectory optimization | |
| Trajectory accuracy (strict, unordered, subset, edit distance) | LangChain AgentEvals | Arize Phoenix |
| EDR, ERP, LCSS distance metrics | AgentEvals docs |
| LLM-as-judge for trajectory quality | OpenAI, Agent evals | Arize Phoenix |
| 8.5 Agent architecture discovery | |
| Evolving complete agent systems | AlphaEvolve, Google DeepMind | GEPA paper, arXiv:2507.19457 |
| ARC-AGI as architecture discovery case | ARC Prize | AlphaEvolve, Google DeepMind |
| 8.6 Skill optimization | |
| Skills as optimizable text artifacts | Anthropic, Claude Code: Best practices |
| Cross-model skill transfer | GEPA paper, arXiv:2507.19457 |
| 8.7 Production systems optimization | |
| Scheduling policy optimization | GEPA paper, arXiv:2507.19457 |
| Routing algorithm evolution | GEPA paper, arXiv:2507.19457 |
| 8.8 Multi-objective and Pareto optimization | |
| Pareto frontier maintenance | GEPA paper, arXiv:2507.19457 |
| Minibatch reflection strategy | GEPA paper, arXiv:2507.19457 |
| 8.9 Optimization infrastructure | |
| Evaluation caching and parallelization | gepa-ai.github.io/gepa |
| CI/CD integration for optimization | Google SRE, Release Engineering | OpenAI, Evaluation best practices |
| 8.11 Anti-patterns | |
| Overfitting to golden sets | Hamel Husain and Shreya Shankar, LLM Evals FAQ | GEPA paper, arXiv:2507.19457 |
| Reward hacking | Anthropic, Demystifying evals for AI agents |
Optimization Algorithm Selection Guide
| Algorithm | Best For | Compute Cost | Generalization | Key Constraint |
|---|---|---|---|---|
| DSPy MIPROv2 | Few-shot examples, module pipelines | Low | Yes | Fixed program topology |
| GEPA | Complex prompts, competing objectives | Medium | Yes | Requires rich ASI |
| optimize_anything | Any text artifact | Medium–High | Configurable | Needs evaluator function |
| AlphaEvolve | Code and algorithms | High | No (single-task) | Significant compute budget |
| OpenEvolve | Open-source code evolution | Medium–High | No (single-task) | Community-maintained |
F.4 Chapter 11 Reference Map (Production)
| Topic in Chapter 11 | Reputed primary sources |
|---|---|
| Why productionizing agents is different | OpenAI, Production best practices; Sculley et al., Hidden Technical Debt in ML Systems; Pan et al., MAP |
| Agent capability reality check | METR, Measuring AI ability to complete long tasks; Pan et al., MAP |
| Picking the right production targets | OpenAI, GDPval; Pan et al., MAP |
| Architecture risks in "more agents" | Cemri et al., Why Do Multi-Agent LLM Systems Fail?; Pan et al., MAP |
| Production readiness as "release unit" | Google SRE, Release Engineering; Google SRE Workbook, Configuration design |
| Evals as gates (offline and pre-prod) | OpenAI, Evaluation best practices; Anthropic, Demystifying evals for AI agents; Pan et al., MAP |
| Observability envelope for agents | OpenTelemetry traces concepts; W3C Trace Context; OpenTelemetry GenAI semantic conventions |
| Canary, rings, staged rollout | Google SRE Workbook, Canarying releases; Google SRE, Reliable product launches |
| Shadow mode / traffic teeing | Google SRE Workbook, Canarying releases; Istio traffic mirroring docs |
| Automated canary analysis | Spinnaker, automated canary analysis guide; Kayenta docs |
| Feature flags and kill switches | Martin Fowler, Feature Toggles; OpenFeature spec |
| Dark launch | Martin Fowler, Dark Launching |
| Online experiments (A/B) | Kohavi et al., Trustworthy Online Controlled Experiments; DORA State of DevOps |
| SLOs and error budgets | Google SRE (error budgets and SLO practice) |
| Incident response and postmortems | Google SRE Book (postmortem culture); Google SRE on-call concepts |
| Overload, backpressure, load shedding | Google SRE Workbook, Identifying and Recovering from Overload |
| Retries, backoff, timeouts | AWS Builders' Library, Timeouts/retries/backoff; OpenAI production best practices |
| Circuit breakers | Martin Fowler, Circuit Breaker pattern |
| Idempotency and safe retries | Stripe, Idempotent requests; AWS guidance on idempotency |
| Compensation and recovery workflows | Microservices.io, Saga pattern; Martin Fowler, Saga |
| Chaos engineering and game days | Principles of Chaos Engineering; AWS Fault Injection Service; Azure Chaos Studio |
| Latency and cost engineering | OpenAI latency optimization; OpenAI prompt caching; OpenAI cost optimization |
| Data readiness and pipeline reliability | Google Cloud MLOps; Sculley et al., Hidden Technical Debt; Google SRE Workbook |
| Security risks for LLM apps | OWASP Top 10 for LLM Applications; NIST AI Risk Management Framework |
| Evidence bundles for governance | NIST AI RMF; OpenAI model optimization |
F.5 Chapter 12 Reference Map (Governance)
| Concept | What it covers | Primary sources |
|---|---|---|
| Multi-tenant policy isolation | Policy versioning, context isolation, approval matrices | Enterprise architecture practices |
| Tenant-aware retrieval | Filtering, validation, audit segregation | Data governance standards |
| Lifecycle gates | When and how to review agents | NIST AI RMF, internal governance practice |
| Evidence packages | What to show at each gate | OpenAI Evals guide, Anthropic engineering practices |
| Risk taxonomy | Categorizing agent risks | OWASP LLM Top 10, ISO/IEC 23894 |
Navigation: ← Appendix E | Appendix Index | Appendix G →