Chapter 3 — Principles for Building Agentic AI

TL;DR

Pick the right problems — use agents only where adaptive, multi-step reasoning adds value over deterministic code, and define tight goals with explicit outcomes and constraints before writing a single prompt.
Start simple and earn autonomy — begin with read-only, human-in-the-loop drafts and promote to higher autonomy only when evaluation evidence and oversight infrastructure justify it.
Make the loop and scaffolds explicit — treat the observe-think-act-reflect cycle as visible, budgeted orchestration and use scaffolds (planner-executor, extract-transform-reason, critic-reviewer) to replace prompt-only control with testable structure.
Context-first, tools-as-contracts — assemble step-specific context like a versioned API, and enforce safety at tool boundaries with typed schemas, policy checks, and idempotency rather than hoping the model behaves.
Evaluate continuously, observe everything, compose small — wire inner-loop runtime checks and outer-loop harness regressions into CI/CD, instrument every run with traces and kill switches, and prefer composable skills over monolithic agents.

Chapter 2 introduced the mechanics of agentic systems. You learned about the agent loop (observe → think/plan → act → reflect), tools as contracts, memory vs context, context engineering, autonomy and human oversight (HITL/HOTL/HOOTL), evaluation (inner and outer loops), scaffolds, skills, and evaluation harnesses.

This chapter is about engineering discipline.

Agentic systems are powerful because they allow a model to decide what to do next and take actions across multiple steps. That flexibility is also what makes them risky and hard to maintain if you don't impose structure.

These principles help you choose problems where agentic AI provides real leverage. They help you design agents that are testable and safe to operate. They prevent "prompt spaghetti" and brittle demos. They help you build systems that can improve over time instead of regressing quietly.

If Chapter 2 is "how agents work", Chapter 3 is "how to build them like an engineer".

flowchart TB
  A[Pick the right problems] --> B[Define a clear goal]
  B --> C[Outcome + constraints]
  C --> D[Start simple<br/>earn autonomy]
  D --> E[Make the agent loop explicit]
  E --> F[Scaffolds shape behavior]
  F --> G[Context-first design<br/>Context engineering]
  G --> H[Oversight by design<br/>HITL / HOTL / HOOTL]
  H --> I[Evaluation is a feature<br/>Inner + outer loops]
  I --> J[UX<br/>Human–agent collaboration]
  J --> K[Explainability]
  K --> L[Harnesses for safe testing]
  L --> M[Tools as contracts]
  M --> N[Memory boundaries]
  N --> O[Observability + kill switches]
  O --> P[Compose patterns and skills<br/>Avoid monoliths]

Figure 3.1: These principles build on each other—start upstream (problem, goal) and work downstream (execution, safety, ops).

3.0 Why Principles?

Traditional software engineering makes certain assumptions. You can reason about behavior from code paths. System state transitions are explicit. Tests cover the majority of meaningful branches. The program does what you wrote, within bugs.

Agentic systems break some of these assumptions. Decision-making is probabilistic due to LLM sampling and context sensitivity. Control flow may be dynamic based on tool choice, number of steps, and branching. State is partially implicit through context assembly, retrieval, memory, and hidden internal reasoning. Failures are often "soft"—plausible but wrong outputs and subtle policy misses.

The goal isn't to eliminate uncertainty. It's to contain uncertainty. You constrain where the agent can act through tool contracts, budgets, and permissions. You validate boundaries with schemas, policy gates, and idempotency. You build feedback loops through evaluation and monitoring. You design UX that keeps humans in control where needed. You make behavior explainable and traceable through traces, evidence, and diffs.

Engineering takeaway

Treat an agent as operational software with probabilistic decisions. Your job is to make that probabilistic part bounded and observable.

Do

Make uncertainty explicit: budgets, stop conditions, "needs review" states.
Assume you will debug runs in production—design for traces and replay.

Don't

Treat prompt quality as the only lever.
Ship without a harness, telemetry, and rollback/kill switches.

HR Leave example: A leave assistant that "usually" gets the right answer is still a risk if it sometimes approves leave without checking balance. Principles exist to prevent dangerous failures: wrong approvals, silent policy misses, and untraceable decisions.

3.1 Select the Right Problems for Agentic AI

Use agents where you need adaptive multi-step behavior, not where you need reliable deterministic execution.

The most expensive failure mode in agentic programs is building a complicated agent where a normal service would have worked better.

What agentic AI is good at

Agentic AI shines when the task requires the system to interpret messy inputs like emails, PDFs, chat transcripts, and screenshots. It excels at handling many variations of a workflow without you enumerating every branch. It's good at gathering information across systems such as knowledge bases, CRM, ticketing, and ERP. It performs well at reasoning under uncertainty with trade-offs, missing fields, and fuzzy matching. It's effective at choosing among actions—which tool to call, what to ask next, and when to stop.

A practical heuristic: the more the work looks like a human operator saying "Let me check this… now I'll look that up… I think the right next step is…" the more likely agentic AI can help.

What agentic AI is not good at

Avoid agentic AI for straight-through deterministic rules. A rules engine is cheaper and safer. Avoid it for tight correctness guarantees without possible verification. Avoid it for high-volume low-value tasks where cost dominates. Avoid it for hard real-time loops with strict millisecond budgets or safety-critical control.

The triage table

Signal	Good fit for Agentic AI	Better as LLM helper	Better as traditional software
Workflow shape	Multi-step, exceptions common	Mostly single-step	Deterministic, stable flow
Inputs	Unstructured/messy	Semi-structured	Highly structured
Decision nature	Judgement/trade-offs	Classify/summarise	Clear rules, exact logic
Risk tolerance	Bounded with guardrails	Low-risk outputs only	Very low tolerance for error
Value per task	High	Medium	Low
Integration	Needs orchestration	Read-only retrieval	CRUD/ETL

Table 3.1: A quick triage to decide whether a use case warrants an agent, a simple LLM helper, or no LLM at all.

Engineering takeaway

Before you write prompts, write down the baseline solution using workflows, scripts, or rules. Identify what portion is messy and variable. Use agentic AI only for that messy portion. Keep deterministic parts deterministic.

Do

Use agents to augment stable workflows instead of replacing them wholesale.
Start with an "LLM helper" version first if unsure (classification, summarization, drafting).

Don't

Use agents as a substitute for missing process definition.
Use agents to mask low-quality data or broken upstream systems.

HR Leave example: A good agentic slice interprets free-text leave requests, asks clarifying questions, retrieves policy snippets, and drafts an approval note. Computing leave balance or writing payroll adjustments should remain deterministic tool calls with strict validation.

3.2 Design Agents with Clear, Specific Goals

The best agents have tight contracts.

A common failure mode is designing an agent like a general chatbot: "Help the HR team" or "Answer any tender questions" or "Be an operations assistant."

That creates ambiguity in three places. First, decision logic: what does "done" mean? Second, context scope: what information is relevant? Everything? Third, evaluation: what is success? It becomes vague.

Instead, treat an agent like a service. Align with Chapter 2's goal/task/policy separation. Give it a clear API-like goal. Define a bounded tool set and step breakdown. Make constraints explicit. Define success criteria and failure handling.

Field	Example
One-line purpose	"Process leave applications according to HR policy and notify stakeholders."
Inputs	Leave request form + employee ID
Outputs	Decision + rationale + drafted notification
Non-goals	"Does not modify payroll; does not grant policy exceptions."
Tools allowed	`get_balance`, `get_policy`, `approve_leave`, `draft_email`
Oversight	Approvals > 2 days = HITL; low-risk approvals = HOTL
Success metrics	% correct vs golden set; escalation rate; cycle time
Failure policy	If low confidence → escalate; never guess missing fields

Table 3.2: A lightweight "agent charter" that forces clarity on scope, tools, oversight, and success metrics.

Why this matters for implementation

A crisp goal enables smaller prompts and smaller context windows. It means fewer tools and lower risk surface area. It enables narrower evaluation sets and faster iteration. It creates clearer UX expectations: "this agent does X, not Y."

It also makes routing and composition easier. You can build a "front door assistant" that routes to specialist skills (in the Chapter 2 sense: reusable capability bundles).

Engineering takeaway

If you can't express the agent's goal as a tight contract, you'll pay for it later in context bloat, tool misuse, and evaluation ambiguity.

Do

Write an agent charter and store it with the codebase.
Reject features that violate the charter unless you deliberately expand scope and update evaluation + oversight.

Don't

Let "nice-to-have" behaviors sneak in without updating evaluation and oversight.
Let the agent's goal drift into "be helpful" without defining what "helpful" means for this workflow.

HR Leave example: A clear goal is: "Given an employee request, approve/reject/escalate based on balance + policy." A vague goal is: "Handle all HR queries." The vague goal will force you to include too much context, expose too many tools, and make correctness impossible to evaluate.

3.3 Outcome-First, Not Model-First

You are building a system to change an operational metric, not a model demo.

Mid-level engineering reality: you will face trade-offs. Correctness versus latency. Cost versus quality. Safety versus autonomy. User convenience versus governance.

If you don't define outcomes early, those trade-offs become political and subjective.

What to define up front

Define your primary metric—the business value you want. This might be time saved, cost reduced, throughput increased, errors decreased, or CSAT improved.

Define secondary metrics—what you're willing to trade off. This includes tokens per task, average latency, and escalation rate.

Define hard fails—what must never happen. Examples include sending emails to the wrong recipient, changing records without approval, and policy violations.

Define budget constraints: cost per case, max tool calls, and SLOs.

Define operating constraints: data residency, PII handling, and audit requirements.

This makes model and architecture choices concrete. If latency is critical, you choose a smaller model with better scaffolding and caching. If correctness is critical, you use more HITL with stronger tool validation and narrower scope. If cost is critical, you use fewer calls with shorter context and reranking or summarisation.

Engineering takeaway

Outcomes and constraints should drive autonomy level, tool exposure, context budgets, and evaluation gates.

Do

Create a "metrics card" per agent (one page).
Use it to guide prompt/context/tool decisions and to justify autonomy upgrades.

Don't

Treat model choice as the main design decision.
Optimize for benchmark scores when your production metric is "time-to-resolution" or "incident rate".

HR Leave example: If the main KPI is cycle time, you may accept more HOTL automation for low-risk annual leave. If the main KPI is policy correctness, you'll keep more HITL and invest in stricter tool contracts plus harness coverage before raising autonomy.

3.4 Start Simple, Then Increase Autonomy

Autonomy is earned through evidence.

Autonomy is not binary. It's a dial you turn up when your system demonstrates reliability and your oversight model can support it. Chapter 2, §2.6 defined the five autonomy levels (L0–L4) and the HITL/HOTL/HOOTL oversight modes. The principle here is: earn each level through evidence.

What "start simple" looks like in practice

Start with read-only tools. Produce drafts and recommendations under HITL. Add automatic actions only for things that are low impact, reversible, and easy to audit.

Then expand. Increase scope. Allow more tools. Allow more write actions. Relax approvals to monitoring under HOTL.

Engineering takeaway

Treat autonomy upgrades like production permission changes. Require evidence, reviews, and rollback plans.

Do

Create explicit "graduation criteria" for autonomy upgrades: quality thresholds, incident rate, audit results, regression pass rates.
Start HOTL with sampling + alerts before moving to HOOTL.

Don't

Use autonomy to compensate for lack of workflow clarity or missing tools.
Make irreversible write tools available before you have harness coverage and observability.

HR Leave example: Start with "Draft an approval note + list policy evidence" under HITL. Only later auto-approve low-risk cases (such as 1 day annual leave with sufficient balance and no blackout dates) under HOTL sampling.

3.5 Make the Agent Loop Explicit

If you can't draw the loop, you can't debug it.

Every agent has an agent loop as defined in Chapter 2. The agent observes state and inputs. It thinks and plans the next step. It acts via tools. It reflects and updates state. Then it stops or repeats.

For mid-level engineers, treat this like any other system that has retries, idempotency concerns, partial failures, side effects, and state transitions.

If the loop is hidden, you lose deterministic recovery. You lose audit trails. You lose observability. You lose safe rollback.

Engineering takeaway

"Agentic" is mostly "runtime orchestration with decisions." Make the orchestration visible.

Do

Implement agent steps explicitly (even if you use a framework).
Record each step as a trace span (step name, inputs, outputs, latency).
Define stop conditions: max steps, max tool calls, max wall time, and escalation rules.

Don't

Allow infinite loops ("keep trying until success") without budgets.
Hide side-effecting tool calls inside opaque chains where you can't intercept/approve.

HR Leave example: A clean loop is: parse request → fetch balance → retrieve policy → decide → propose → (approve?) → write → verify → stop. If the agent "keeps checking policy" indefinitely, you'll burn tokens and still end up escalating. Add step and time budgets.

3.6 Shape Behaviour with Scaffolds, Not Just Prompts

Scaffolds reduce randomness by adding structure.

A scaffold (as defined in Chapter 2) is the structure around the model that enforces stages, schemas, checklists, and separation of concerns between planning and acting or between extracting and deciding.

Scaffold type	When it helps	Typical structure
Checklist / template	Consistency, completeness	Prompt sections + required fields
Planner–executor	Multi-step tasks, tool orchestration	Plan call → step loop
Extract–transform–reason	Messy inputs	Extract → normalise → decide
Critic / reviewer	Reduce errors	Generate → critique → revise
Router + specialists	Multiple capabilities	Route → specialist agent/skill

Table 3.3: Common scaffolds you can mix and match to improve reliability and testability.

flowchart LR
  O[Observe<br/>inputs, state] --> X[Extract / Normalize]
  X --> P[Plan]
  P --> A[Act<br/>tool call]
  A --> R[Reflect<br/>check result]
  R -->|repeat| P
  R -->|done| F[Finalize<br/>output + explanation]

Figure 3.2: A scaffolded loop creates clear "checkpoints" for validation, oversight, and evaluation.

Engineering takeaway

Scaffolds turn "LLM behavior" into something closer to "software behavior"—bounded, testable, and composable.

Do

Use structured outputs and typed schemas early.
Insert checks between phases (validation, policy checks, human approvals).
Make budgets part of the scaffold (max steps, max tool calls, timeouts).

Don't

Treat prompt-writing as the only control mechanism.
Combine planning + execution + explanation into one giant prompt when you need auditability.

HR Leave example: Use an Extract → Normalize scaffold to turn "next Fri half day" into {start_date, end_date, duration, leave_type}. Then a Plan → Execute scaffold ensures you never approve before get_balance and get_policy checks succeed.

3.7 Context-First Design (Context Engineering)

Most agent failures are context failures.

In Chapter 2 language, memory is what you store and context is what you pass. Context-first design treats context assembly as a first-class subsystem.

Key engineering practices include context recipes per step (not one global prompt), context builders in code as pure functions or components, budgets for tokens, latency, and cost, and structure through schemas, sections, and delimiters.

This is also where RAG lives in practice. Retrieval isn't "a feature"—it's part of context construction.

Engineering takeaway

Treat context like an API surface. Version it, test it, and review changes like code.

Do

Version your context recipes like code (changes require eval).
Keep policy/system instructions separated from user content.
Include provenance: what was retrieved, from where, and why it was selected.

Don't

Mix instructions and retrieved content without clear delimiters (prompt injection risk).
Treat "just add more documents" as a quality strategy.

HR Leave example: A good context recipe includes the current task ("decide leave approval"), balance snapshot, retrieved policy clauses (blackout dates, approval thresholds), and tool contracts. It does not include the entire HR policy handbook or months of chat history.

3.8 Oversight by Design (HITL/HOTL/HOOTL)

Oversight is part of system design, not a business process afterthought.

Using the oversight modes from Chapter 2, §2.6 (HITL/HOTL/HOOTL), define oversight per action class:

Action class	Examples	Typical autonomy	Oversight	UX pattern
Read-only	search KB, summarise	L0–L2	HOOTL	Inline result + sources
Low-impact write	tags, internal notes	L1–L2	HOTL	Activity feed + undo
Medium-impact	status changes, approvals within limits	L2–L3	HOTL / HITL	Propose→confirm or sampling
High-impact	money, compliance, irreversible writes	L1–L2	HITL	Approval screen + evidence

Table 3.4: Oversight must be designed at the granularity of action types, not as a single global rule.

Enforce oversight at tool boundaries using approval tokens, dual-control, role-based permissions, and "two-person rule" patterns. Build oversight into UX through review screens, diffs, evidence panels, and audit feeds.

Engineering takeaway

Oversight is a product and platform feature. You implement it in tool contracts, UI affordances, and monitoring workflows.

Do

Treat each tool as an action surface with explicit risk classification.
Make "approval required" enforceable in code, not only a prompt instruction.

Don't

"Let the agent decide" whether approval is needed.
Assume monitoring is enough when the action is irreversible or high impact.

HR Leave example: Auto-approving a 1-day annual leave might be HOTL with sampling. Rejecting medical leave, approving exceptions, or changing records in a system-of-record should be HITL with an approval screen that shows balance and policy evidence.

3.9 Evaluation is a Feature, Not a Phase (Inner + Outer Loops)

Without evaluation, you can't safely iterate.

Chapter 2 separates the inner loop (per-run checks and behaviors) from the outer loop (improvement over many runs through datasets, regression suites, and prompt/tool updates).

Evaluation happens at two levels:

Dimension	Inner loop evaluation (runtime)	Outer loop evaluation (offline/continuous)
Purpose	prevent bad actions now	improve system over time
Checks	validation, policy, safety filters	golden sets, judge scoring, regression suites
Signals	tool errors, low confidence	success rate, drift, cost/latency, quality
Output	allow/block/escalate	prompt/tool/policy updates

Table 3.5: Inner loop evaluation is about containment; outer loop evaluation is about improvement.

flowchart TB
  subgraph INNER[Inner loop: runtime]
    Req[User event] --> Agent[Agent loop]
    Agent --> Guard[[Runtime checks<br/>validation, policy, safety]]
    Guard --> Tool[(Tools and APIs)]
    Tool --> Agent
    Agent --> Out[Outcome]
  end

  Out --> Trace[(Traces and logs)]

  subgraph OUTER[Outer loop: improvement]
    Trace --> Harness[[Harness runs<br/>golden sets, regressions]]
    Harness --> Metrics[(Metrics and reports)]
    Metrics --> Updates[Prompt, tool, policy updates]
  end

  Updates --> Agent

Figure 3.3: Inner-loop checks prevent bad actions immediately; outer-loop harness runs turn real behavior into improvements.

Roadmap note

Chapter 5 shows how to build harnesses that run scenarios and capture traces. Chapter 6 defines what to measure (metrics, judges, and datasets). Chapter 7 shows how to operationalise evaluation in CI/CD and in production.

Engineering takeaway

Treat evaluation as a dependency of autonomy. No eval means no safe upgrades.

Do

Treat evaluation as part of your CI/CD gates for agents.
Add test cases whenever you discover a failure mode (bug → test → fix).
Track regressions per dimension: correctness, safety, cost, latency, UX.

Don't

Change prompts/tools in production without regression evaluation.
Rely only on "human spot-checks" without measurable baselines.

HR Leave example: Inner loop: block approval if balance_check failed or policy retrieval returned "unknown". Outer loop: maintain a golden set of past leave cases (blackouts, overlaps, insufficient balance) and run it on every prompt/tool/policy change.

Singapore assurance lens: In Singapore, IMDA and the AI Verify Foundation are building shared testing guidance and tooling for GenAI applications. This includes the Global AI Assurance Pilot, Project Moonshot, and the Starter Kit for Safety Testing of LLM-based Applications. Treat evaluation as an engineering deliverable: repeatable tests, documented evidence, and metrics you can share with risk owners, auditors, and regulators.

3.10 Design for Human–Agent Collaboration (UX)

Good agent UX is mostly about control and recovery.

Agent UX should answer five questions. What can this agent do? What is it doing right now? What changed? Why did it decide that? How do I override it?

flowchart LR
  U[User goal] --> Q{Clarify needed?}
  Q -->|yes| C[Ask clarifying questions] --> P[Proposal + evidence]
  Q -->|no| P[Proposal + evidence]
  P --> D{Approve / Edit / Reject}
  D -->|Approve| A[Execute tools]
  D -->|Edit| E[User edits] --> A
  D -->|Reject| R[No action + feedback]
  A --> S[Outcome + audit trail + undo]

Figure 3.4: "Ask → Show → Act" is a safe default UX pattern, especially for write actions.

UX moment	What user needs	Minimal UI element
Before action	What will happen?	Plan preview ("I will…")
At decision	Why this outcome?	short rationale + evidence
During execution	What's happening now?	step indicator / activity log
After execution	What changed?	diff / audit trail + undo
When uncertain	What now?	"needs review" + escalate

Table 3.6: UX and explainability are the same problem viewed from the user's perspective.

Engineering takeaway

UX is part of safety. Your UI should expose autonomy, evidence, and recovery paths. Otherwise the agent becomes a black box.

Do

Make autonomy visible ("draft-only", "auto for low-risk cases").
Provide undo and escalation as first-class actions.
Show evidence for decisions (sources, policy snippets, tool outputs).

Don't

Hide high-impact actions behind vague "Done" messages.
Require users to open logs to understand what changed.

HR Leave example: When the agent proposes approval, show dates, leave type, balance used, policy clause, and the exact change that will be written. If the user edits dates or type, the agent should re-run checks and update the proposal before any write.

3.11 Design for Explainability

Explainability isn't one thing—it's different depth for different personas.

Users don't need chain-of-thought. They need a rationale in a few bullets, evidence from policy snippets and sources, and confidence or risk cues.

Reviewers and auditors need more. They need step traces, tool calls, and context/prompt versions.

flowchart TB
  U[End user] --> E1[Short rationale<br/>2-3 bullets]
  R[Reviewer] --> E2[Rationale + evidence<br/>clauses, snippets]
  A[Auditor<br/>Ops] --> E3[Trace view<br/>steps, tools, versions]
  E1 --> E2 --> E3

Figure 3.5: Progressive disclosure lets you be explainable without overwhelming everyone.

Engineering takeaway

Explanations must be grounded in system truth—retrieved sources, tool outputs, and policies. They should not be post-hoc storytelling.

Do

Persist "decision records": decision + reasons + evidence IDs + tool results.
Build "Explain this" UI that surfaces evidence with provenance.
Record prompt/context versions so auditors can reproduce behavior.

Don't

Present rationales that can't be traced back to evidence.
Treat explainability as only a UI problem—without traces, you can't explain reliably.

HR Leave example: A good explanation cites "Annual leave balance: 8.0 days" and "Policy §X: approvals required for >2 days." A bad explanation is "Approved because it seems reasonable" with no balance or policy evidence.

3.12 Test and Contain Agents with Harnesses

Harnesses (Chapter 2 vocabulary) let you run realistic scenarios repeatedly. You can simulate tools safely. You can compute metrics consistently. You can catch regressions before production.

A good evaluation harness includes a scenario library with golden sets and edge cases. It includes sandbox or fake tools for safe side effects. It includes automated scoring using rules and judges where needed. It exports to dashboards for trending over time.

Engineering mindset: treat harness work like unit tests, integration tests, and load tests for agent behavior.

Engineering takeaway

A harness is the practical bridge between the outer loop and production confidence.

Do

Start small: 20–50 scenarios covering the core workflow + common failures.
Add scenarios whenever you fix a bug or handle an incident.
Run harness in CI for every change to prompts, tools, context recipes, or autonomy.

Don't

Use only ad-hoc UI testing as your release gate.
Test against production systems when you can use sandbox/fakes.

HR Leave example: Your harness should include cases like overlapping leave, blackout dates, insufficient balance, half-day requests, and "missing leave type". Run these cases any time you change parsing, retrieval, or approval thresholds.

3.13 Tools as Contracts, Not Suggestions

Tools are your safety boundary (Chapter 2: tools as contracts). They should validate inputs with schemas. They should enforce permissions and policy. They should be idempotent where possible. They should log side effects. They should return structured errors so the agent can recover or escalate.

Treat tool design like API design. If your tool API is sloppy, agent behavior becomes sloppy.

Engineering takeaway

Tool contracts are where you enforce safety. They are not where you "hope the model behaves."

Do

Use typed schemas (e.g., Pydantic) and strict validation.
Put policy checks in the tool layer (not only in prompts).
Make error messages structured and actionable (error codes + remediation hints).

Don't

Expose "do anything" tools (arbitrary SQL, arbitrary HTTP) without strong constraints.
Let the model invent parameters without schema validation.

HR Leave example: Make approve_leave(request_id, approver_id) require a valid approver role, a policy-check token, and an idempotency key. If the agent tries to approve without meeting prerequisites, the tool should reject and force escalation—not "best effort".

3.14 Memory With Boundaries (Memory vs Context)

Memory increases capability. It also increases privacy risk, prompt injection and exfiltration risk, and debugging complexity.

Aligning to Chapter 2: memory is storage and context is what you pass. A strong default for engineering teams is to store operational traces for audit and debug. Store semantic knowledge separately in RAG corpora and indexed content. Store user personalization only with clear consent and TTL. Never store secrets in retrievable memory.

Engineering takeaway

Memory is a data system. Treat it like one with schemas, retention, access control, and deletion workflows.

Do

Separate stores: traces vs long-term knowledge vs user memory.
Implement TTL and deletion workflows (user/tenant "forget me").
Redact or avoid storing sensitive fields by default.

Don't

Store raw prompts + sensitive documents indefinitely "for convenience."
Mix tenants/domains in a way that allows cross-leakage.

HR Leave example: You may store "pending manager approval for request #123" with TTL. You should not store medical diagnosis details or raw documents in long-lived retrievable memory unless policy explicitly allows it.

3.15 Design for Observability and Kill Switches

You need per-run traces showing step-by-step execution. You need aggregate dashboards showing quality, cost, latency, and escalations. You need kill switches to disable agent, disable tool, or disable per-tenant.

Agents are operational software. Treat them like you treat any system with side effects. You need "stop the bleeding" controls.

Engineering takeaway

If you can't answer "what happened on this run?" you can't operate agents safely.

Do

Trace every tool call (args, outputs, latency, errors) and every loop step.
Track budgets: tool calls, tokens, retries, wall time.
Implement kill switches at multiple levels: agent-wide, tool-specific, tenant-specific.

Don't

Rely only on unstructured logs.
Ship without a way to disable side-effecting tools quickly.

HR Leave example: If a policy update causes a spike in wrong approvals, you need a one-click switch to disable auto-approval and revert to HITL drafts while you investigate.

3.16 Prefer Composable Patterns (and Skills) Over Monolithic Agents

Monolithic agents are hard to test and govern. Prefer composition. Use router plus specialists patterns. Use planner/executor scaffolds. Use critic/reviewer scaffolds. Use reusable context builders. Use shared harnesses. Use reusable skills—capability bundles with instructions, tools, and small orchestration.

This is normal software engineering. Composition beats giant classes.

Engineering takeaway

Skills and scaffolds let you scale capability without scaling risk and complexity linearly.

Do

Design your system as a set of small components with clear contracts.
Reuse and version skills the way you version APIs.
Test components independently, then as an end-to-end workflow in the harness.

Don't

Put every behavior into one agent prompt and hope evaluation catches regressions.
Duplicate similar prompt/tool logic across agents instead of extracting a skill.

HR Leave example: Build separate skills for parsing requests, policy retrieval, balance checks, decisioning, explanation, and execution. That makes it easy to upgrade only the parsing skill (like better date handling) without destabilizing approvals.

3.17 Putting It All Together

When you apply these principles to something like an HR leave agent, you get several things. You get a narrow goal and bounded tools. You get explicit approvals for high-impact actions under HITL. You get scaffolded steps with validation and budgets. You get context recipes per step through context engineering. You get continuous evaluation via a harness in the outer loop. You get UX that shows plan, evidence, and allows override. You get explainability grounded in traces and evidence. You get observability and kill switches in production.

That's the difference between an impressive demo and a reliable system.

Engineering takeaway

The "agent" is the visible tip. The scaffold, tool contracts, context engineering, harness, and observability are what make it production-grade.

Do

Treat each new capability as: design → implement → harness → monitor → iterate.
Keep autonomy conservative until metrics and incidents say otherwise.

Don't

Treat "it works on my demo" as production readiness.

3.18 Quick Design Checklist

Problem fit
Goal clarity (goal/task/policy)
Outcomes + constraints
Autonomy + oversight per action class (HITL/HOTL/HOOTL)
Explicit agent loop
Scaffolds for structure + budgets
Context recipes + builders (context engineering)
Evaluation (inner + outer) + harness
UX: Ask → Show → Act + recovery
Explainability: rationale + evidence + traces
Harness runs + regressions
Tools: typed, validated, policy-enforcing contracts
Memory: scoped + TTL + access control (memory vs context)
Observability + kill switches
Composition + skills over monoliths

3.19 Key Takeaways

Right problem, right tool — Use agentic AI only for the messy, judgement-heavy portion of a workflow and keep deterministic logic deterministic.
Tight contracts beat broad ambition — Define each agent's goal, non-goals, tools, oversight level, and success metrics in a concise charter before you build anything.
Autonomy is earned, not assumed — Start at the lowest autonomy level with human-in-the-loop review and graduate upward only when evaluation data, incident rates, and audit results justify it.
Scaffolds over prompts — Structure agent behaviour with explicit loops, phase gates, budgets, and typed schemas rather than relying on prompt engineering alone.
Context engineering is the real lever — Treat context assembly as a first-class, versioned subsystem with per-step recipes, token budgets, and clear provenance for every retrieved snippet.
Evaluation is continuous, not occasional — Wire inner-loop runtime checks (validation, policy, safety) and outer-loop harness regressions (golden sets, scoring, drift detection) into every release gate.
Observe everything, kill anything — Instrument every run with structured traces and aggregate dashboards, and maintain kill switches at agent, tool, and tenant granularity so you can stop the bleeding in seconds.

Next --> Chapter 4 shows how these principles become concrete in scaffold patterns — the reusable inner-loop structures that shape agent behaviour.

3.20 Reference Map

Principle	What it covers (engineering angle)	Primary sources (clickable)
Selecting the right problems	When agents add leverage vs when deterministic software wins	Anthropic — Building effective agents (Dec 2024) Chip Huyen — Agents (Jan 2025)
Clear goals and tight scope	"Agent charter", non-goals, bounded actions, success criteria	Anthropic — Building effective agents (Dec 2024) NIST AI 600-1 — GenAI Profile (Jul 2024, PDF)
Outcome-first design	Define metrics + constraints before model choice	OpenAI — How evals drive the next chapter in AI for businesses (Nov 2025) OpenAI — Evaluation best practices (Docs)
Start simple → earn autonomy	Autonomy levels, phased rollout, graduation criteria	Anthropic — Building effective agents (Dec 2024) NIST AI 600-1 — GenAI Profile (PDF)
Make the control loop explicit	Step logging, retries, idempotency, side effects	Chip Huyen — Agents (Jan 2025) OpenAI — Working with evals (Docs)
Scaffolds over prompt-only	Planner/executor, extract→plan→act→reflect structures	Anthropic — Building effective agents (Dec 2024) Chip Huyen — Agents (Jan 2025)
Context-first design (context engineering)	Context budgets, step-specific context builders, delimiting	Anthropic — Effective context engineering for AI agents (Sep 2025) Anthropic — Introducing Contextual Retrieval (Sep 2024)
Retrieval/RAG details	Chunking, grounding, retrieval failure modes	Anthropic — Introducing Contextual Retrieval (Sep 2024) Jay Alammar — RAG patterns & best practices (InfoQ, May 2024)
Oversight by design (HITL/HOTL/HOOTL)	Approval gates, override paths, accountability	NIST AI 600-1 — GenAI Profile (PDF) Azure — AI agent orchestration patterns (Jul 2025) Google PAIR — People + AI Guidebook
Evaluation is a feature (inner + outer loops)	Runtime checks + offline harness + regression gates	OpenAI — Working with evals (Docs) OpenAI — Evaluation best practices (Docs) OpenAI — How evals drive… (Nov 2025) IMDA — Starter Kit for Safety Testing of LLM Apps (May 2025, PDF) AI Verify Foundation — Global AI Assurance Pilot (Feb 2025) AI Verify Foundation — Project Moonshot
Harnesses for safe testing	Golden sets, sandbox tools, repeatable regressions	OpenAI — Working with evals (Docs) AI Verify Foundation — Moonshot (GitHub) Anthropic — Effective harnesses for long-running agents (Nov 2025)
Tools as contracts	Typed schemas, validation, policy enforcement	OpenAI — Function calling (Docs) OpenAI — Structured outputs (Docs) Anthropic — Writing effective tools for agents (Sep 2025)
Memory boundaries	What to store, TTL, privacy, retrieval risks	Anthropic — Effective context engineering… (Sep 2025) NIST AI 600-1 — GenAI Profile (PDF)
Observability + kill switches	Traces, dashboards, disable agent/tool/tenant, sandboxing	Anthropic — Beyond permission prompts… (Oct 2025) OpenAI — Evaluation best practices (Docs)
Composable patterns and skills	Router+specialists, multi-agent systems, reusable skills	Anthropic — Building effective agents (Dec 2024) Anthropic — Multi-agent research system (Jun 2025) Anthropic — Agent Skills (Oct 2025)
UX: human–agent collaboration	Control, recovery, transparency, expectation setting	Google PAIR — People + AI Guidebook Nielsen Norman Group — AI as a UX Assistant (Oct 2023) Azure — AI agent orchestration patterns (Jul 2025)

Table 3.7: Reference map from Chapter 3 principles to reputable sources and primary documentation.