03 · Expert
The production and systems concepts. This is the tier that separates a demo from a service real people rely on — reliability, cost at scale, security, and knowing when not to reach for AI at all. Where you've built something related, a Related implementations line links to your write-up in Learnings.
Multi-agent orchestration tradeoffs
Multi-agent means several specialised agents cooperating — e.g. analyzer → drafter → reviewer — instead of one general agent doing everything.
The upside: each agent has focused context, so it's more accurate; bugs localise ("the analyzer got it wrong" beats "the giant prompt got something wrong"); you can eval each stage independently.
The cost: more orchestration, more tokens, more places for the chain to break, and — the subtle one — context fragmentation: no single agent sees the whole picture, so information the drafter needed but the analyzer discarded is just gone.
The live debate you should know for interviews: Anthropic's Building Effective Agents argues for multi-agent when warranted; Cognition's Don't Build Multi-Agents argues most teams overcomplicate and a single well-prompted agent with good context wins. The disagreement is the lesson. Start single-agent / workflow. Move to multi-agent only when you have evidence — evals showing a single agent can't reach the quality — not on speculation. Both essays are in the reading list.
A lower-risk middle ground: multi-phase but not multi-agent — specialised steps coordinated by code and sharing state through a database rather than talking to each other. The shared store is the context, which sidesteps fragmentation.
Prompt caching
Providers let you cache the static prefix of a prompt so you don't re-pay (in tokens and latency) to process it on every call. If your system prompt + reference docs are 8k tokens and identical across calls, caching them means each call only pays for the new part.
Why it matters at scale: for RAG and agent systems, the same big system prompt / instructions / examples get resent constantly. Caching can cut cost and latency materially. Structure prompts so the stable part comes first (cacheable) and the variable part comes last.
Security & guardrails
More power (tools, autonomy, data access) means more risk (rogue actions, data leakage, prompt injection). Defence-in-depth:
- Deterministic guardrails — hard rules outside the model that it cannot override: "block any write to prod," "require confirmation over $100." The model advises; code enforces.
- Reasoning-based defences — a small "guard model" screens inputs and proposed actions for injection and policy violations before they execute.
- Agent identity & least privilege — each agent gets a verifiable ID and only the permissions it needs, so a compromised agent's blast radius is contained. The strongest version of this: simply don't wire up the most dangerous capability at all (e.g. let the system draft an action but never execute it).
- Input authentication — verify a request is actually from who it claims (e.g. HMAC signature verification on webhooks, using a constant-time compare to avoid timing attacks).
See also the Kaggle agents notes for the governance/identity angle at scale.
Graceful degradation
Production systems fail partially — an API times out, a parse fails, a rate limit hits. Graceful degradation is designing so that when a piece fails, the system recovers or fails safely and visibly, rather than crashing or corrupting state.
The core techniques:
- Retries with exponential backoff — transient failures often succeed on a second try a moment later. (A structured-output parse failure is often worth one retry, e.g. at a slightly higher temperature.)
- Idempotency — running the same operation twice has the same effect as once, so a
retry can't double-apply (e.g.
INSERT ... ON CONFLICT DO NOTHING). - Atomic state + rollback — never leave the system in a half-updated state; if a step fails, unwind cleanly.
- Visible failure — when you can't recover, tell the user. Silent failure is the worst outcome because no one knows to act.
Related implementations: V2 drops the JSON-parse retry → silent stall — a case where removing the retry and failing silently regressed both halves of this concept at once.
MCP
MCP (Model Context Protocol) is an open standard (from Anthropic, 2024) for plugging tools and data sources into agents. Instead of hand-writing a bespoke integration for every service, a tool speaks MCP and any MCP-aware agent can use it. MCP connects agents to tools — the sibling standard A2A connects agents to other agents.
Why it matters: it's becoming the USB-C of agent tooling. Learning to consume and expose MCP servers is how you integrate without reinventing an adapter each time. The bespoke adapters you hand-write today (fetch a record, search a store, post a comment) are exactly the kind of thing MCP standardises — and exposing them as an MCP server lets other agents reuse them.
Scaling
Making a system handle more — more requests, more users, more data — without falling over. The concerns shift from "does it work" to "does it work under load":
- Latency vs throughput — response time for one request vs requests handled per second. Different problems; streaming helps perceived latency, not throughput.
- Rate limits & backpressure — provider APIs cap calls/minute; a system at scale must queue and shed load gracefully rather than hammer and fail.
- Concurrency & locking — when many requests touch the same state, races appear. Atomic state transitions can double as locks.
- Stateless vs stateful — stateless request handlers scale horizontally; state has to live somewhere shared (a DB), not in process memory. Choosing the right state model early gets you most of this almost for free.
When not to use AI
The most senior skill in the book. AI is the wrong tool when:
- A deterministic solution exists. If rules or a lookup table solve it, use them — they're cheaper, faster, testable, and don't hallucinate.
- Errors are unacceptable and uncatchable. If there's no human check and a wrong answer is catastrophic, an inherently-probabilistic system is a poor fit.
- You can't evaluate it. If you have no way to measure whether output is good, you can't improve or trust it — build the eval first, or don't build it.
- The cost/latency doesn't pay for itself. An LLM call that's slower and pricier than the thing it replaces, for no quality gain, is theatre.
The discipline that runs through Anthropic's essay: start with the simplest thing that
works. Don't reach for an agent when a workflow will do; don't reach for a workflow
when one prompt will do; don't reach for a prompt when an if statement will do. The
best designs use AI for the genuinely fuzzy judgment and keep deterministic machinery
everywhere determinism is available.
Human-in-the-loop
HITL is deliberately designing where a human approves or corrects the agent before
it takes a significant or irreversible action. It's both a tool
(ask_for_confirmation()) and a design stance.
Two reasons it matters:
- Trust — even a 95%-right agent has a 5% that can be catastrophic in production. The human checkpoint catches it.
- Adoption — people won't use a tool they don't trust. Visible approval gates build that trust over time.
Good HITL has the human reviewing the right things (not everything, not nothing), makes the review fast (a 30-second yes/no, not a 30-minute audit), and gives the reviewer context for the decision. The pattern that scales: be fast and autonomous where speed is safe, and stop for a human wherever the action is consequential. Placement is the art — depth in Component 6 of the AI-First Methodology.
Where this leaves you
You now have the full arc: foundations → intermediate → expert. The book is not finished — it grows every time you build something and bring back a learning: new theory gets researched and added, and your implementations get linked under Related implementations. When a topic here has no learning pointing at it yet, that's simply work you haven't reported yet.
Go build. Then come back and tell me what happened.