Skip to content

Table of Contents

The Agentic AI Builder's Handbook

Principles, Patterns, Scaffolds, and Harnesses

Version: 1.1

Part I — Orientation

  1. Why Agentic AI, Why Now
    • What is agentic AI and why the timing is right
    • Decision frameworks for agent-appropriate problems
    • The execution gap: from demo to production

Part II — Foundations: Concepts and Principles

  1. Core Concepts of Agentic AI

    • The agent loop: observe → plan → act → reflect
    • Tools as contracts, memory vs context
    • Autonomy levels and human oversight models

  2. Principles for Building Agentic AI

    • Context-first design
    • Oversight by design
    • Evaluation as a feature

Part III — Design: Scaffolds, Harnesses, and Patterns

  1. Scaffolds: Structuring Agent Behaviour

    • Inner loop patterns: ReAct, Plan-Execute, state machines
    • Skills, routing, and composition
    • Checkpoint and recovery patterns

  2. Harnesses: How We Test and Observe Agents

    • Evaluation harness architecture
    • Offline vs online evaluation
    • Evidence bundles and release gates

Part IV — Evaluation & Optimization

  1. Evaluation Foundations: Metrics, Judges, and Data

    • Metric taxonomy and rubrics
    • LLM-as-judge patterns
    • Dataset design and golden sets

  2. Evaluation in Practice: Tooling, Patterns, and Playbooks

    • Open-source evaluation frameworks
    • Lifecycle playbooks and gate policies
    • Continuous improvement loops

  3. Optimization: From Evaluation to Improvement

    • Three optimization modes: single-task, multi-task, generalization
    • Actionable Side Information (ASI) and reflective evolution
    • Trajectory, architecture, skill, and production system optimization

  4. Best Practices and Design Patterns

    • Prompts, tools, and context engineering
    • Memory patterns and orchestration
    • Model fallback and cost optimization

Part V — UX, Productization, and Whole-System Design

  1. UX for Agentic AI Applications

    • Autonomy UX and progressive disclosure
    • Evidence, confidence, and transparency
    • Error handling and recovery patterns

  2. From Prototype to Production

    • Deployment patterns and rollout strategies
    • Observability and monitoring
    • Resilience, cost management, and scaling

Part VI — Governance & Organizational Readiness

  1. Governance, Policy and Lifecycle

    • Decision rights and policy frameworks
    • Lifecycle gates and checkpoints
    • Multi-tenant and enterprise considerations

  2. Teams, Skills and Operating Models

    • Team structures and roles
    • Capability building and maturity models
    • From projects to platform capabilities

Part VII — Case Studies and Patterns in the Wild

  1. Case Studies
    • HR Leave Agent (primary example)
    • IT Help Desk Agent
    • Invoice/Procurement Assistant
    • Failure patterns and lessons learned

Appendices

Appendix Index — Quick access to all appendix materials

A. Production Challenges, Checklists and Templates - Why production is hard - Challenge categories and mitigations - Failure flow diagram - Ecosystem landscape - Agent design checklist - Production launch checklist - Incident response checklist - Evaluation planning checklist - Governance one-page template - Cost engineering checklist - Case study template - Failure prevention checklist

B. HR Leave Agent — Complete Reference - Agent overview and architecture - Production patterns and walkthroughs - Tool, memory, and RAG patterns - Evaluation harness configuration - Governance artifacts and PR checklist

C. HR Leave Agent — Prompts and Evaluation Rubrics - System prompt and step-specific prompts - LLM judge rubrics (eligibility, safety, citation, quality) - Composite judge configuration - Critic/verification prompt

D. HR Leave Agent — Synthetic Datasets - Dataset structure and manifest - Golden set cases across leave types - Edge cases and adversarial cases - Regression cases from incidents - Dataset generation scripts

E. HR Leave Agent — Code Reference - State machine scaffold implementation - Tool patterns and idempotent writes - Evaluation harness code - Interactive demo and companion repository

F. Evaluation Quick-Reference and Source Map - Metrics registry template - Gate policy matrix template - Synthetic data worksheet - Chapter 7 concept-to-source reference map

G. Agent Extension Points — Tools, Skills, Hooks, MCP, and Subagents - Extension point comparison (tools, skills, hooks, MCP, subagents) - Tool design principles and read vs write patterns - Skills: anatomy, design, and when to use - Hooks: lifecycle callbacks, guardrails, and telemetry - MCP: standardised integration architecture - Subagents: delegation patterns and trade-offs - Composition patterns and decision framework - HR Leave Agent end-to-end example

H. Cross-Reference Index - Topic to chapter matrix - Chapter dependency map - Case study to pattern map - Quick start by task - Quick start by role

I. Glossary - 23 key terms and definitions

Quick Reference

Reading Paths by Role

Role Core Chapters Reference Time
Product Managers 1, 3, 10, 11, 14 Ch 12, 13, 14.6 ~3 hours
Software Engineers 2, 4, 5, 9, 11 Ch 6-7, App B ~5 hours
Architects 2, 4, 9, 11, 12 App A, Ch 14 ~4 hours
Governance/Risk 1, 3, 6, 12, 14 Ch 11, App A ~4 hours
ML/AI Engineers 2, 6, 7, 8, 9, 11 Ch 4, 5 ~5 hours
Engineering Managers 1, 3, 11, 13, 14 Ch 12, App A ~4 hours

Reading Paths by Task

Task Chapters
Decide if we should build an agent Ch 3 §3.1, Ch 11 §11.0.4, Ch 14.6
Design an agent from scratch Ch 4, Ch 9, App B
Evaluate an existing agent Ch 6, Ch 7, Ch 5
Optimize an agent Ch 8, Ch 7
Take an agent to production Ch 11, App A
Set up governance for agents Ch 12, App A
Understand why our agent failed Ch 14.6, Ch 11.4

Reading Paths by Experience

If you're... Start with... Then...
New to agents Ch 1–3 (Foundation) Ch 4–5 (Design)
Building now Ch 4–7 (Design → Eval) Ch 8–9 (Optimize → Patterns)
Shipping soon Ch 8–12 (Optimize → Prod → Gov) Ch 14 (Cases)
Debugging issues Ch 14.6 (Failures) Ch 11.4 (Observability)

See Reading Paths for detailed guidance.