Chapter 6 — Evaluation Foundations: Metrics, Judges, and Data
"If scaffolds define how the agent thinks, evaluation defines what 'good thinking' looks like."
TL;DR
- Start every evaluation effort by translating business goals and SME expertise into concrete, levelled metrics rather than chasing model-centric scores.
- Measure agents across six dimensions -- task success, quality, safety, latency, cost, and UX -- at step, task, session, and system levels to pinpoint where failures originate.
- Combine four evaluator families -- deterministic rules, reference-based checks, LLM-as-judge with calibrated rubrics, and human/critic-agent review -- into scorecards with hard gates and soft thresholds.
- Build versioned evaluation datasets from golden sets, production traces, and synthetic/adversarial scenarios so your test suites grow with every incident and policy change.
- Treat evaluation as a continuous, designed capability -- not a one-time QA step -- feeding production observations back into regression suites and improvement loops.
With harnesses in place (Chapter 5), you need to decide what to measure and how to judge it. This chapter provides the foundations. Chapter 7 turns them into operational playbooks.
We tackle three core questions:
- How do you turn business goals into metrics that make sense for agents?
- What kinds of judges and evaluators can you use, and how do you combine them?
- How do you build datasets, benchmarks, and synthetic scenarios that let you test agents safely and thoroughly?
Running example: HR Leave Agent (see Chapter 4 for introduction, Chapter 14.2 for complete case study)
6.0 From Harnesses to Metrics: Where Evaluation Fits
With harnesses in place (Chapter 5), you need to decide what to measure and how to judge it. Harnesses capture spans, traces, and telemetry—but they do not tell you what is good or bad. Without carefully defined metrics and judges, you end up with beautifully instrumented agents that you still cannot trust.
We follow a simple path:
- Start from business goals and subject matter experts (SMEs).
- Translate those into metrics at different levels.
- Decide how to judge each metric (rules, LLMs, humans, critic agents).
- Collect and shape the data you run through your harnesses: logs, traces, golden sets, synthetic scenarios.
Chapter 7 will operationalise this: tools, CI gates, monitoring, drift checks, and playbooks.
6.0.1 GenAI evaluation as software testing for probabilistic systems
If you come from traditional software engineering, it helps to treat GenAI evaluation as the testing discipline for probabilistic systems.
The intent is familiar. You want confidence that changes do not break the product. The difference is what you are testing. In deterministic software, behavior is largely governed by code paths. In agentic systems, behavior is governed by a moving bundle: prompts, context, retrieval corpora, tool contracts, policies, model versions, and runtime settings. Even if the code is unchanged, the system can shift because the "inputs to reasoning" shifted.
That is why this handbook treats evaluation as a designed capability, not a final QA step. It is the mechanism that makes a probabilistic system shippable and operable.
What stays the same
Several testing instincts transfer directly. You still want fast checks on every change for PR gating. You still want regression protection against known failures. You still want progressive rollout through shadow mode, canary releases, and rollback triggers. You still want observability to debug and to detect drift.
What changes
The "shape" of testing changes because your system has variance, subjective dimensions, and heavy data dependence.
| Dimension | Traditional Software Testing | GenAI / Agentic AI Testing |
|---|---|---|
| System behavior | Deterministic | Probabilistic; run-to-run variance |
| Oracle of truth | Spec + expected outputs | Rubrics + SMEs + reference corpora (context-grounded) |
| Unit of test | Function/class/module | Prompt+context→response, multi-turn workflows, tool calls |
| Pass/Fail | Binary assertions | Graded judgments (faithfulness, relevance, safety), thresholds |
| Data dependence | Low/moderate | High (retrieval corpora, prompts, tools, policies) |
| Failure modes | Logic/IO/latency | Hallucination, bias, jailbreaks, tool misuse, drift |
| Coverage | Code paths | Task & context coverage (gold sets, adversarial sets) |
| Regression | Deterministic replay | Stochastic replay + judge consistency checks |
| Observability | Logs/metrics | Spans, traces, reasoning artifacts, tool-call transcripts |
| Governance | Test reports | Evidence packs: alignment, safety, data handling, risk tiers |
Two practical implications follow.
First, you cannot rely on a single "golden output" for many tasks. Instead, you define what "good" means using SME rubrics and scorecards. Then you enforce thresholds and hard gates. For example, policy violation rate must be zero.
Second, reproducibility becomes a spectrum. You still use deterministic validators wherever possible. But you also expect some flakiness in model judgment and generation. That is why you calibrate judges, version judge prompts, and use multiple runs or consistency checks for sensitive gates.
📚 Test pyramid mapping: For a detailed mapping of classic software testing concepts (unit, integration, acceptance, regression, staging) to their agent evaluation analogues, see Chapter 5, Table 5.0.
Chapter 6 defines the measurement primitives (metrics, evaluators, datasets). Chapter 7 turns those primitives into operational discipline: CI gates, rollout patterns, monitoring, and improvement playbooks.
Figure 6.1: The evaluation ladder mirrors modern release discipline
flowchart TB
C[Change introduced<br/>prompt, retriever, tool, policy, scaffold] --> U[Fast deterministic checks<br/>schemas, invariants, tool contracts]
U --> R[Offline regression harness<br/>golden set + risk suites]
R --> J[Judges and critics<br/>rubrics + scorecards]
J -->|hard gate fail| B[Block merge or rollback<br/>attach evidence bundle]
J -->|passes| S[Shadow mode / canary rollout<br/>tight thresholds]
S --> M[Monitoring + sampling<br/>drift and safety triggers]
M --> F[Failure analysis<br/>add new golden cases]
F --> RKey point: for agentic systems, "tests" are not only pre-release checks. They are a continuous loop. Production traces become evaluation data. Incidents become permanent regression cases. That is the practical reason evals matter.
6.1 From Business Goals to Metrics (SME-First)
6.1.1 Start from Business KPIs, Not Model Scores
It is tempting to start with model-centric metrics like "task success rate should be ≥ 85%" or "hallucination rate should be low" or "we want a higher RAG faithfulness score."
Those metrics are useful, but they are proxies. They only matter if they support the business outcome and the organisation's risk tolerance. Risk-oriented frameworks (for example NIST's GenAI Profile) push you to tie evaluation to risk management objectives rather than leaderboard scores.
For the HR Leave Agent, stakeholders usually care about four things. They want HR workload reduction, meaning fewer tickets HR staff must handle manually. They want consistency and compliance, meaning answers match HR policy, agreements, and labour law. They want good employee experience, meaning people do not feel lost, frustrated, or treated unfairly. They want low risk, meaning no disputes, grievances, or regulatory trouble because of wrong decisions.
A simple pattern:
Business objective → Evaluation goal → Concrete metrics
Example 1:
- Business objective: "Reduce manual HR effort on leave queries by 30% within 6 months."
- Evaluation goal: Agent handles routine leave requests correctly without HR intervention.
- Metrics: containment rate, HR escalations per 100 interactions, HR agreement rate on sampled decisions.
Example 2:
- Business objective: "Maintain zero labour law breaches in leave approvals and rejections."
- Evaluation goal: No decisions that contradict policy or law.
- Metrics: policy or legal violation rate (target: 0), number of overturned decisions after review.
Keep each metric tied to a "why" the business recognises. This keeps optimisation anchored when you later tweak prompts, swap models, or adjust scaffolds.
Table 6.1: Business objectives to metrics (HR Leave Agent)
| Business objective | Evaluation goal | Primary metrics | SME owners (examples) |
|---|---|---|---|
| Reduce HR workload on routine leave queries | Maximise self-serve resolution without HR intervention | Containment rate; HR escalation rate; HR time saved | HR Ops lead; Shared services |
| Maintain zero labour law breaches | Ensure decisions comply with policy and law | Policy or legal violation rate; audit exceptions; overturn rate | Legal or compliance; HR policy owner |
| Improve employee trust and satisfaction | Deliver clear, correct, explainable outcomes | CSAT; dispute rate; explanation quality score | EX lead; HRBP |
6.1.2 Levels of Metrics
Agentic systems have multiple layers of behaviour. You need metrics at matching levels.
Step-level metrics evaluate one operation at a time (span-by-span). Did the retrieval pull the right policy clauses? Was the tool call query correct and the result plausibly used? Did the model follow the expected schema?
Task-level metrics evaluate the outcome of a workflow (end-to-end). Was the leave request correctly approved or rejected? Did the system-of-record end up in the right state? Was the correct person notified?
Session or user-level metrics evaluate the user journey. Did the user get what they needed? Did they abandon or escalate? Did they rate the interaction positively?
System-level metrics aggregate across time and population. What proportion of cases are handled by the agent vs humans? How many safety incidents occur per month? Are metrics improving or drifting?
You do not need dozens of metrics at each level. But you do want at least one or two per level. This helps you locate failures: in retrieval, in a decision, or in the UX.
Table 6.2: Levels of metrics
| Level | Question it answers | Typical metrics | HR Leave example |
|---|---|---|---|
| Step-level | Did this operation work correctly? | Retrieval precision@k; tool success; parsing accuracy | HRIS call success rate |
| Task-level | Was the workflow outcome correct? | Decision correctness; end state validity; compliance | Correct approve or reject decision |
| Session-level | Did the user get what they needed? | CSAT; escalation; time to resolution | Employee CSAT for leave queries |
| System-level | Is the system behaving well overall? | Incident rate; drift; dispute rate; coverage | Disputes per 1,000 decisions |
Figure 6.2: Metric stack (where metrics attach)
flowchart TB
S["Step-level<br/>(span)"] --> T["Task-level<br/>(workflow outcome)"]
T --> U["Session-level<br/>(user journey)"]
U --> Y["System-level<br/>(fleet over time)"]6.1.3 SME-Led Metric Definition
Most important metrics cannot be defined by engineers alone. SMEs understand what counts as a correct decision. They know which mistakes are annoying versus catastrophic. They know where the legal, regulatory, or contractual boundaries are.
For the HR Leave Agent, SMEs might include HR operations, HR business partners, legal or compliance, and union representatives (if applicable).
A practical pattern is a metric design workshop:
- Ask SMEs in plain language: "What does good look like?" "What is unacceptable?"
- Capture phrases without jargon.
- Translate into precise metrics and thresholds.
- Validate the mapping with SMEs.
Key idea:
SMEs define the semantics of correct and safe. Engineers turn those semantics into measurable metrics.
6.2 Taxonomy of Metrics for Agentic Systems
Now that metrics are anchored in business goals and SME input, you can use a reusable taxonomy.
6.2.1 Task Success and Correctness
These metrics ask: did the agent achieve the right outcome?
There are two types. Exact success applies to structured tasks with one correct answer. Graded success applies when partial credit makes sense, especially for explanations.
HR Leave examples:
- decision_correct (yes/no)
- approver_correct (yes/no)
- system_update_correct (0–2: wrong, partial, full)
For high-risk decisions, treat correctness as binary: acceptable or not. For explanations and guidance, graded scales capture "good enough" versus "needs work".
6.2.2 Quality and Usefulness
Even when the decision is right, the interaction can still be poor.
Quality metrics commonly include faithfulness (does the answer stick to retrieved or policy sources?), completeness (does it cover key points the user needs?), clarity and structure, and tone.
HR Leave examples:
- explanation_faithfulness
- completeness_score (0–2)
- tone_appropriateness
6.2.3 Safety and Robustness
These metrics ask: did anything bad happen?
Common dimensions include policy and legal compliance, privacy and security, fairness and non-discrimination, abuse and harmful content, and robustness under unusual or adversarial inputs (prompt injection, etc.).
HR Leave examples:
- policy_violation (yes/no, target: 0)
- legal_violation (yes/no, target: 0)
- pii_leakage (yes/no, target: 0)
- fairness_flag (yes/no or score)
Safety metrics are often non-negotiable. Any violation is a hard failure.
6.2.4 Latency and Reliability
Latency metrics include P50, P95, and P99 response time for first response and end-to-end completion.
Reliability metrics include tool failure rates, retry rates and degraded mode behaviour, and "stuck" behaviour such as loops, repeated clarifying questions, and dead-ends.
6.2.5 Cost and Efficiency
Cost metrics include cost per interaction, cost per completed task, and cost per correct outcome.
Cost becomes actionable when compared to baseline human effort. It also becomes actionable when tied to specific levers: context size, number of calls, tool usage, and review sampling.
6.2.6 UX and Satisfaction Metrics
Explicit signals include ratings, thumbs, and feedback forms. Implicit signals include abandonment, rephrasing, escalation, and undo actions.
UX metrics are often leading indicators. They drift before correctness drops visibly.
Table 6.3: Metric taxonomy and when it matters
| Metric category | Definition | Example metric | When it is critical |
|---|---|---|---|
| Task success | Correct outcome | Decision correctness rate | Automation, audits |
| Quality | Usefulness beyond correctness | Explanation completeness | Trust, adoption |
| Safety | Avoid harmful or non-compliant outcomes | Policy violation rate | Regulatory and reputational risk |
| Latency | Speed of response and completion | P95 time to decision | Peak periods, UX |
| Cost | Resource and review spend | Cost per correct decision | Scale and budget |
| UX | Satisfaction and behaviour signals | CSAT, escalation rate | Rollout health |
📚 Related sections: - Cost metrics → Chapter 11, §11.6 - UX metrics → Chapter 10, §10.12 - User research metrics → Chapter 7, §7.5
6.3 Evaluators: Judges, Critics and Rules
Metrics tell you what you want to measure. Evaluators decide how you assign scores.
You will usually combine four families:
- Deterministic rules and validators
- Automatic reference-based metrics
- LLM-as-judge
- Human and critic-agent evaluation
6.3.1 Deterministic Checks and Rule-Based Validators
These are explicit checks that do not require machine learning. They include schema validation, type checks, business rules, and precisely encodable policy rules.
HR Leave examples: - Reject outputs where leave start > leave end. - Reject outputs where requested days > remaining leave balance. - Reject outputs where mandatory documentation is missing.
These checks are cheap and interpretable. They are excellent for catching scaffold and tool integration failures. They cannot judge nuance such as tone, fairness, or "reasonable interpretation".
6.3.2 Reference-Based Automatic Metrics
These metrics compare the agent's output to a reference. They work well when there is a clear reference answer, for example an approver ID or a labelled decision.
They have limitations. They are hard to maintain for open-ended tasks. They are weak on tone and structure. They do not reliably explain why answers differ.
6.3.3 LLM-as-Judge
Use one model to evaluate outputs against a rubric. Provide the input, agent output, relevant context, and rubric. Ask the judge to score each rubric item. Parse scores into structured fields.
Treat it like a model, not an oracle. It can be biased or inconsistent. Calibrate against SME labels. Re-audit periodically and after major model or prompt changes.
6.3.4 Human Evaluation and SME Panels
Humans remain the gold standard for defining rubrics and failure modes, labelling seed sets, auditing automated judges over time, and reviewing borderline or high-impact cases.
In high-stakes domains, SMEs should sign off on what you measure, what thresholds are acceptable, and when autonomy is allowed versus when human approval is required.
6.3.5 Critic Patterns as Evaluators
A critic evaluator is a specialised agent whose job is to evaluate, not generate the primary answer.
For HR Leave, you might have a Policy Critic that outputs: - a score for policy compliance - violated clauses (if any) - a short rationale
Treat critic evaluators similarly to LLM judges. Calibrate with SME labels. Audit for drift. Keep scope narrow.
Table 6.4: Evaluator selection guide
| What you are judging | Best first choice | Why | Common failure mode |
|---|---|---|---|
| JSON schema, dates, balances | Deterministic rules | Cheap and unambiguous | Rules drift when policy changes |
| Structured outcomes (approver ID) | Reference match | Clear ground truth | Ground truth stale |
| Policy compliance with text | Judge with citation requirement | Handles nuance | Judge inconsistency without calibration |
| Tone and empathy | Judge with rubric | Rubric-based grading | Rubric mismatch to culture and context |
| Prompt injection resistance | Red-team suite + security rules | Adversarial coverage | Missing attacker creativity |
| End-to-end workflow success | Mixed: reference + rules + audit | Multi-signal | "Looks right" but wrong system state |
6.3.6 Combining Signals: Scorecards and Thresholds
Combine rules, reference metrics, LLM judges or critics, and human labels.
Build a scorecard with primary metrics, supporting metrics, hard constraints (any violation fails), and soft metrics (tolerable within a band).
These thresholds become launch gates, rollback triggers, and criteria for when to allow autonomy versus require human review.
Table 6.5: Practical scorecard template
| Metric | Level | Evaluator(s) | Gate type | Owner |
|---|---|---|---|---|
| Policy or legal violation rate | Task, system | Rules + SME audit + judge | Hard | HR + Legal |
| PII leakage rate | Task, system | Rules + red-team suite | Hard | Security |
| Decision correctness | Task | SME labels + reference match + calibrated judge | Primary | HR Ops |
| Retrieval quality (precision@k) | Step | Reference-based | Supporting | Engineering |
| Explanation faithfulness | Task | Judge with policy excerpt | Supporting | Product |
| P95 time to decision | Task, system | Telemetry | Soft | Engineering |
| Tool failure rate | Step, system | Telemetry | Soft | Engineering |
| Escalation rate | Session, system | Telemetry | Supporting | HR Ops |
Figure 6.3: Evaluator stack (how scores are produced)
flowchart TB
H[Human SMEs and panels] -->|Calibrate rubrics| J[LLM-as-judge and critic evaluators]
J --> SC[Scorecard and gates]
R[Reference-based metrics] --> SC
V[Rules and validators] --> SC📚 Related sections: - SME roles in teams → Chapter 13, §13.1 - Governance involvement → Chapter 12, §12.2 - HR Leave rubric example → Chapter 14, §14.2
Practitioner callout: Singapore assurance lens
Singapore's GenAI assurance work encourages a concrete, evidence-driven framing. Examples include IMDA's LLM safety testing starter guidance and the AI Verify Foundation ecosystem. These cover risk dimensions such as hallucination, undesirable content, data disclosure, and vulnerability to adversarial prompts. This is a helpful lens for defining hard gates and for structuring safety suites in enterprise deployments.
📚 Related sections: - Graders in harnesses → Chapter 5, §5.4 - LLM judge implementation → Chapter 7, §7.2 - Human-in-the-loop evaluation → Chapter 7, §7.5
6.4 Building Evaluation Datasets, Benchmarks and Task Suites
6.4.1 Sources of Evaluation Data
Typical sources include: - product requirements and user stories - SOPs and policy documents - historical logs and tickets (anonymised) - production traces and bugs - red-team and risk scenarios
6.4.2 Designing Golden Sets with SMEs
📋 Example Dataset: Appendix D provides a complete synthetic dataset for the HR Leave Agent, including golden set cases across leave types (§D.2), edge cases for policy conflicts and ambiguous requests (§D.3), adversarial cases for prompt injection and PII extraction (§D.4), regression cases from past incidents (§D.5), and a generation script for creating additional cases (§D.6).
A golden set is a small, carefully curated collection of cases with high-quality labels.
Golden sets need three characteristics. They need coverage across the cases you care about. They need clarity so labels are unambiguous. They need stability, meaning they are versioned with documented changes.
6.4.3 Using Spans, Traces and Logs (from Ch5)
Traces give you rich raw material. You can transform a trace into a task-level case (input, final action(s), final state). You can derive multiple step-level test items, one per span.
Minimum fields worth retaining for evaluation (even if you store more):
- trace_id, span_id
- scenario_id and suite_id
- model_id, prompt_version, judge_prompt_version
- policy_corpus_version
- retriever_config_version
- tool_version (per connector)
- decision and final_state_hash (or equivalent "system-of-record" state marker)
This is how observability becomes an evaluation data factory rather than just a debugging aid.
Figure 6.4: From traces to evaluation items
flowchart LR
A[Retrieve policy] --> B[Parse certificate] --> C[Check balance] --> D[Select approver] --> E[Update HRIS]
T[Whole trace] --> TT[Task-level test case]
A --> SA[Step-level item]
B --> SB[Step-level item]
C --> SC2[Step-level item]
D --> SD[Step-level item]
E --> SE[Step-level item]6.4.4 Simulation and Synthetic Data for Evaluation
Simulation matters for several reasons. It enables deliberate edge-case coverage. It lets you rehearse policy changes. It lets you explore tail risks safely. It helps you avoid exposing PII.
Common patterns include:
- Template-based generators
- Scaffold-powered scenario generators
- Counterfactuals from real traces
- Adversarial simulations (prompt injection, data disclosure attempts, tool misuse)
Security-focused taxonomies (for example OWASP guidance for LLM applications) are practical sources of adversarial categories to include in red-team suites.
Figure 6.5: Simulation harness flow
flowchart LR
G["Scenario generator<br/>(templates + LLM)"] --> C["Synthetic cases"]
C --> E["Simulated environment<br/>(employees, balances, policies)"]
E --> A["Agent in harness<br/>(end-to-end runs)"]
A --> V["Evaluators<br/>(rules, judge, audits)"]
V --> R["Simulation report<br/>(failure heatmap)"]6.4.5 Dataset Governance
Treat datasets like code. Apply versioning and changelogs. Write documentation (lightweight "datasheets"). Track coverage. Enforce PII and access controls.
Make governance concrete by tying dataset versions to what actually changes agent behaviour: - policy corpus version - prompt and context recipe version - retriever and reranker config version - tool schema and connector version - judge prompt and rubric version
Table 6.6: Dataset "datasheet" fields (minimum viable)
| Field | What to record |
|---|---|
| Purpose | What decisions this dataset supports (gates, regressions, audits) |
| Coverage | Scenario types, employee types, leave types, risk categories |
| Label source | SME, rules, judge, hybrid |
| Known gaps | What is missing or underrepresented |
| Versioning | Semver, changelog, policy version mapping |
| Privacy | PII handling, anonymisation method, access rules |
| Intended use | Offline eval, plus any online sampling or auditing |
6.4.6 From Golden Sets to Benchmarks and Task Suites
A task suite is a curated collection of cases for a specific aspect. A benchmark is a stable, versioned suite tracked over time.
Example suites for HR Leave: - Standard Leave Suite - Edge and Legal-Risk Suite - Red-Team Suite
📚 Related sections: - Golden set management → Chapter 7, §7.3 - Jurisdiction stratification → Chapter 14, §14.2.7 - Incident-to-dataset loop → Chapter 11, §11.4
6.5 Summary and Bridge to Practice
In this chapter you built the foundations. Start with business goals and SMEs, not model scores. Define metrics at multiple levels and dimensions. Combine evaluators: rules, references, LLM judges, humans, critics. Build evaluation data: golden sets, simulation, task suites, versioned and governed.
If scaffolds (Ch4) define how your agent thinks, and harnesses (Ch5) define how you test it, then evaluation foundations (Ch6) define what it means for that thinking to be good enough.
In Chapter 7, you will move from concepts to practice. You will learn how to integrate evaluation into CI/CD and production monitoring. You will learn how to run regressions, canaries, and shadow mode with clear gates. You will learn how to design playbooks for different agent types and risk tiers.
6.6 Key Takeaways
- Business goals first, model scores second -- Every metric should trace back to a business objective and an SME-defined notion of "correct" and "safe"; proxy scores like BLEU or generic accuracy are meaningless without that anchor.
- Metrics live at four levels -- Step, task, session, and system-level metrics each answer a different diagnostic question; you need at least one or two at every level to locate whether a failure is in retrieval, reasoning, tooling, or the overall user experience.
- Six dimensions form a complete taxonomy -- Task success, quality, safety, latency, cost, and UX cover the full surface area of agentic behaviour; safety metrics are often non-negotiable hard gates where any violation is a failure.
- No single evaluator is sufficient -- Deterministic rules catch structural errors cheaply, reference checks verify factual outputs, LLM judges handle nuance at scale, and human SMEs remain the gold standard for rubric definition and periodic auditing.
- LLM judges must be calibrated, not trusted blindly -- Treat an LLM-as-judge like a model: calibrate it against SME labels, version its prompts, re-audit after model or rubric changes, and pair it with consistency checks for sensitive gates.
- Evaluation data is a living asset -- Golden sets, synthetic scenarios, and adversarial suites should be versioned, governed, and continuously expanded from production traces, incidents, and policy changes.
- Evaluation is continuous, not a launch gate -- Production traces feed back into regression suites, incidents become permanent test cases, and scorecards with hard and soft thresholds drive both release decisions and runtime rollback triggers.
Next --> Chapter 7 turns these foundations into operational playbooks -- tooling, CI/CD integration, and repeatable evaluation patterns.
6.7 Reference map
| Concept | What it covers (engineering angle) | Primary sources (clickable) |
|---|---|---|
| Foundations: why evals exist and how to define "good" | ||
| Evaluation is a feature (design-time commitment) | Treating evals as a first-class product and engineering requirement, not last-mile QA | OpenAI - Evaluation best practices |
| SME-first metrics and rubrics | Using domain experts to define "correct" and "unsafe", then translating that into measurable criteria and thresholds | Hamel Husain - LLM Evals FAQ Simon Willison - Creating a LLM-as-a-Judge |
| Business objective → eval goal → metric mapping | Keeping evaluation aligned to business outcomes, not proxy model scores | OpenAI - Evaluation best practices Hamel Husain - LLM Evals FAQ |
| Metrics: what to measure (and where) | ||
| Multi-level metrics (step, task, session, system) | Measuring at the right layer: span-level failures vs end-to-end outcomes vs user experience vs fleet health | OpenAI - Trace grading OpenAI - Agent evals |
| Task success and correctness | Evaluating whether the workflow landed the right outcome (binary gates for high-risk tasks, graded for softer tasks) | Anthropic - Demystifying evals for AI agents OpenAI - Agent evals |
| Quality and usefulness | Faithfulness, completeness, clarity, and style checks that drive user trust and adoption | OpenAI - Evaluation best practices Hamel Husain - LLM Evals FAQ |
| Safety and robustness metrics | Measuring harmful or unacceptable outcomes and treating some as hard constraints (must be zero) | Anthropic - Demystifying evals for AI agents OpenAI - Understanding prompt injections |
| Latency and reliability | P50/P95 timing, tool failure rates, retries, and "stuck/loop" behavior that breaks workflows | OpenAI - Trace grading OpenAI Agents SDK - Tracing |
| Cost and efficiency | Cost per interaction, cost per completed task, and cost per correct outcome to manage economics | Anthropic - Building effective agents OpenAI - Evaluation best practices |
| UX signals and feedback loops | Explicit feedback (ratings) and implicit feedback (rephrases, abandon, escalation) as eval data | OpenAI - Evaluation best practices Chip Huyen - Common pitfalls when building generative AI applications |
| Evaluators: how to score (rules, refs, judges, humans) | ||
| Deterministic checks and validators | Schema checks, business rules, and invariants as cheap, interpretable graders | Anthropic - Demystifying evals for AI agents OpenAI - Evaluation best practices |
| Reference-based metrics and golden answers | Exact match, tolerances, and reference comparisons for tasks with stable ground truth | OpenAI - Working with evals |
| LLM-as-judge fundamentals | Rubric-based judging, structured outputs, calibration, and known failure modes | Hamel Husain - Using LLM-as-a-Judge Simon Willison - Creating a LLM-as-a-Judge |
| Critique-first judging and binary outcomes | Critique shadowing, pass/fail gates, and capturing rationales you can reuse for judge prompts | Simon Willison - Creating a LLM-as-a-Judge Hamel Husain - LLM Evals FAQ |
| Human evaluation and audits | Where humans remain essential: rubric definition, seed labeling, and periodic audits of automated graders | Chip Huyen - Common pitfalls when building generative AI applications Hamel Husain - LLM Evals FAQ |
| Critic evaluators (specialized critics) | Purpose-built critics (policy critic, safety critic, tone critic) to score one dimension well | Anthropic - Demystifying evals for AI agents Hamel Husain - Using LLM-as-a-Judge |
| Mixing graders (rules + refs + LLM + humans) | Combining multiple evaluators, handling disagreements, and building scorecards with hard/soft gates | Anthropic - Demystifying evals for AI agents OpenAI - Agent evals |
| Trace-first evaluation: grading and debugging multi-step systems | ||
| Trace grading | Scoring an entire agent trace and/or individual steps to localize failures | OpenAI - Trace grading |
| Traces and spans as eval substrate | Capturing tool calls, generations, and guardrails as structured artifacts you can replay and grade | OpenAI Agents SDK - Tracing |
| Error analysis and failure taxonomy | Turning failures into buckets, prioritizing fixes, and converting incidents into regression cases | Hamel Husain - LLM Evals FAQ |
| Datasets and suites: what you run through the harness | ||
| Golden sets | Small, high-trust labeled datasets used for regressions and launch gates | OpenAI - Evaluation best practices Anthropic - Demystifying evals for AI agents |
| Synthetic and simulated scenarios | Generating edge cases, counterfactuals, and rare risk cases when real data is sparse or sensitive | Hamel Husain - LLM Evals FAQ OpenAI - Evaluation best practices |
| Adversarial and prompt-injection suites | Designing red-team scenarios for instruction hijacking and untrusted content boundaries | OpenAI - Understanding prompt injections |
| Dataset flywheel (continuous improvement) | Using production traces and reviews to continuously grow eval sets and reduce regressions | OpenAI - Evaluation best practices OpenAI - Agent evals |
| Monitoring: keeping eval meaning stable over time | ||
| Drift and monitoring for eval health | Watching for input and outcome shifts, plus alert fatigue and false positives | Chip Huyen - Data Distribution Shifts and Monitoring |
| Operational execution: running evals programmatically | ||
| Operationalizing eval runs via API | Programmatic eval execution, datasets (JSONL), and result analysis to compare versions | OpenAI - Working with evals OpenAI - Evals API reference |
| Observability and trace standards: OpenTelemetry ecosystem | ||
| OpenTelemetry and vendor-neutral context propagation | Trace context propagation across services and vendors | W3C Trace Context |
| Shipping telemetry (protocol) | Wire protocol used to export telemetry to collectors/backends | OpenTelemetry - OTLP specification |
| Collector as the ingest/control plane | Central pipeline for sampling, redaction, enrichment, fan-out to backends | OpenTelemetry Collector (GitHub) |
| GenAI semantic conventions | Standard span/attribute vocabulary for LLM calls, embeddings, tool use | OpenTelemetry - GenAI semantic conventions |
| OpenInference (AI-native conventions on OTel) | OTel-compatible conventions and instrumentations for AI apps (chains, agents, tools) | Arize - What is OpenInference? |
| Trace backends (examples) | Where traces land and how you explore them (UIs, storage backends) | Jaeger Zipkin Grafana Tempo |
| Metrics exposition standard | Standard for exporting metrics (useful for scorecards and SLO-like gates) | Prometheus - OpenMetrics 2.0 spec |
| OTel-native LLM observability examples | Concrete mappings of AI observations to OTel traces | Langfuse - Tracing data model Arize Phoenix - Instrument tracing |
| Data lineage standard (adjacent) | Dataset and run provenance events for pipelines that generate eval data | OpenLineage |
| Optimization: improving the system using eval metrics | ||
| Evaluation flywheel | Analyze failures → expand evals → iterate changes with measurable deltas | OpenAI Cookbook - Evaluation flywheel |
| Evals-driven prompt optimization | Systematic prompt iteration against a target metric | OpenAI Cookbook - Optimize prompts |
| Eval-driven improvement ladder | Practical ordering of levers to improve a system with evals | OpenAI Cookbook - Eval-driven system design |
| GEPA (reflective prompt evolution) | Metric-guided prompt evolution using reflective mutations and selection | OpenAI Cookbook - Self-evolving agents (GEPA section) arXiv - GEPA |
| Program-level optimization (LM programs) | Optimizing an entire multi-step program against a downstream metric (no per-module labels) | DSPy - Optimizers overview DSPy - MIPROv2 Hugging Face - Paper page (2406.11695) |
| Retrieval evaluation you can optimize | Retrieval metrics and workflows for measuring and improving RAG subsystems | OpenAI Cookbook - Evaluate RAG with LlamaIndex |
| RAG improvement example | Worked example improving RAG behavior using evals and system changes | OpenAI Cookbook - Fine-tuning for RAG with Qdrant |
| Fine-tuning and RFT references (optional, often out of scope) | Guidance on model fine-tuning and reinforcement fine-tuning driven by graders | OpenAI - Distillation guide OpenAI - Fine-tuning best practices OpenAI - Reinforcement fine-tuning |
| Prompt iteration tooling (vendor) | Practical prompt iteration workflows with comparisons and evaluation runs | Anthropic Docs - Eval tool |