Skip to main content

02 · Intermediate

The builder concepts — how the foundations get assembled into real systems, and the knobs you turn to make them good. This is the tier where "it works in a demo" becomes "it works." Where you've built something related, a Related implementations line links to your write-up in Learnings.

RAG design & tuning​

Why RAG exists covered the what. The intermediate skill is that retrieval quality is a system you tune, not a thing that either works or doesn't. The levers:

  • What you embed. Embedding the wrong text is the most common silent failure. If you embed a document's title but the useful signal is in its body, retrieval quietly underperforms and you blame the model.
  • Top-K. How many chunks you retrieve. Too few misses relevant context; too many dilutes the prompt with noise and costs tokens.
  • Similarity threshold. Below some score, a "match" is just the nearest of a bad bunch. Deciding when to return nothing is part of the design.
  • What you put in the payload. Store alongside each vector whatever the generation step needs — labels, metadata, the source text — so you don't have to re-derive it.

Measure retrieval separately from generation. A bad final answer can come from bad retrieval (wrong documents) or bad generation (right documents, wrong use). If you only score the final answer you can't tell which. This is the single most important RAG-tuning habit.

Chunking strategies​

Before you embed long documents, you split them into chunks. How you split governs retrieval quality more than almost anything else.

  • Too big: each chunk covers many ideas; a query matches it for the wrong reason, and the prompt fills with irrelevant text.
  • Too small: ideas get severed from their context; a chunk retrieves but doesn't carry enough to be useful.
  • Overlap: letting chunks share a bit of boundary text avoids splitting a sentence or idea clean in half.
  • Semantic / structural chunking: split on natural boundaries (headings, functions, records) rather than a blind character count.

A useful reframe: chunking is only a problem when your source units are bigger than one coherent idea. If each source document is already one coherent unit, embedding it whole is a legitimate strategy.

Vector database tradeoffs​

A vector database stores embeddings and does fast nearest-neighbour search. The real choice is hosted service vs extension on a database you already run.

Dedicated vector DB (Pinecone, Qdrant, Turbopuffer)Postgres + pgvector
StrengthPurpose-built, scales to huge corpora, rich filteringOne database for both your data and vectors; one thing to run/back up
CostAnother service to host, pay for, secureFree extension on infra you already have
WhenMillions of vectors, heavy filtering, dedicated search teamYou already run Postgres and your corpus is modest

For most application-scale projects, "boring Postgres + pgvector" wins — one fewer moving part beats marginal search performance you won't notice. Reach for the specialised store when scale forces it, not before.

Related implementations: pgvector over Qdrant for V2 — collapsing two datastores into one by moving vectors into an already-running Postgres.

Evals​

An eval is a repeatable test of output quality. Because LLM output is probabilistic, assert output == expected doesn't work — you need to score how good an output is, at scale.

The standard toolkit:

  • Golden dataset — a curated set of inputs with known-good outcomes. Your source of truth.
  • LLM-as-judge — a strong model scores outputs against a written rubric. How you evaluate past what you can eyeball by hand. Have a human validate the judge before trusting it.
  • Metrics that correlate with usefulness — pick measures that track "is this actually good," not what's easy to compute.

Why this is the highest-leverage skill: without evals, every prompt change is a guess and every "it got better" is a vibe. With them, every change is verified and regressions get caught. A widely-repeated finding: comprehensive evaluations often matter more than the initial prompt. Depth on this lives in the reading list (Hamel Husain's eval posts).

Agent patterns​

The vocabulary of how LLM systems are structured. You compose these; they're not mutually exclusive.

ReAct (Reason + Act)​

The workhorse loop: the model alternates reasoning ("I need the order status") and acting (calls a tool), feeding each observation back in, until done. Most single agents are a ReAct loop.

State machines​

Instead of letting the model roam, you model the process as explicit states with allowed transitions, and the LLM only does the work within a state. This trades flexibility for control, predictability, and debuggability — you always know exactly where a task is and what can happen next. A common, powerful refinement: back the state in a database and make each transition an atomic update, so the transition doubles as a lock and races resolve to a single winner.

Tool use / function calling​

You expose functions to the model (get_weather, search_db). The model doesn't run them — it requests a call, your code runs it, you feed the result back. Tools are how an LLM reaches beyond its training data into the live world. A structured contract (JSON schema, or MCP) makes tool calls reliable.

Cost management​

Every call costs tokens, and tokens cost money. Cost optimisation is real engineering, not an afterthought. The main levers:

  • Model routing — cheap model for easy steps. Usually the biggest win.
  • Prompt caching — stop re-paying for the static prefix of a prompt.
  • Token discipline — cap and trim what goes into the window; don't send the whole repo when a slice will do.
  • Fewer calls — a workflow that does the job in 2 calls beats an agent that wanders for 8.

You can't manage what you don't measure — log token usage and cost per call from day one.

Prompt versioning​

Prompts are code. They belong in version control, in files, reviewable — not retyped from memory each time. When a prompt change moves your evals, you want to know which change and be able to roll it back. Splitting prompts by task or phase lets you tune one without disturbing the others, and the diff tells the story. The build recipe treats this as its own step: see Building an Agentic System.

Context engineering​

The successor to "prompt engineering": deliberately curating everything in the context window for each call — instructions, retrieved facts, examples, history, memory — so the model has exactly what it needs and no more.

The failure modes are symmetric:

  • Too much → the signal drowns in noise, cost rises, latency rises, and the model fixates on the wrong thing.
  • Too little → it's missing what it needs and hallucinates to fill the gap.

Most "AI doesn't work" experiences are really "wrong context" experiences. This is often described as the #1 applied-AI skill. The discipline is asking, per step, what does this step actually need to see? — and excluding the rest. The methodology angle is Component 5 of the AI-First Methodology.

Observability​

You can't fix what you can't see. Observability is being able to inspect what your system did in production: logs, metrics, and especially traces — a step-by-step record of an agent's trajectory that answers "why did it do that?".

  • Logs — what happened.
  • Metrics — how much / how often / how fast / how expensive.
  • Traces — the full path of a single request, call by call. The debugging tool.

Tooling: Langfuse, OpenTelemetry. Instrument early — retrofitting observability after a production incident is painful.

Multi-model orchestration​

Using more than one model in a single system, each for what it's best at — the generalisation of model routing. Beyond cheap-vs-strong, you might use a dedicated embedding model, a small classifier, and a frontier reasoner in one pipeline. Pushing this further — many specialised agents rather than many models — is where multi-agent orchestration begins, and that's an expert-tier decision with real tradeoffs.

The provider adapter layer​

A thin module of your own between the app and the vendor's SDK — one function per capability. Nothing else in the codebase touches the SDK.

def call_chat(messages, model): ...    # -> the model's text
def call_embed(texts): ...             # -> vectors

What it buys you:

  • Swap providers in one file — model names and vendors change often.
  • One place for the operational concerns — timeouts, retries, backoff, model choice, cost logging.
  • A test seam — fake the adapter and the suite runs with no network, key or bill.
  • One place to handle failure — every call is a network call to a service that is sometimes slow, rate-limited, or down.

Watch out:

  • What varies per service belongs in config, not code — which key is accepted, which features work, what the model is called. These differ even between two models behind the same URL.
  • Keep it thin. Isolate the SDK; don't re-implement it. Leave an escape hatch (raw response access) for the cases your interface doesn't cover.

Industry practice: standard, and treated as the foundational design decision for production AI — apps talk to an interface you control, never to a vendor directly. Hosted gateways (LiteLLM, Portkey, OpenRouter) are the same idea at infrastructure level, adding routing and fallbacks. Our hand-rolled call_chat / call_embed module is the normal starting point. — Building production AI workflows without vendor lock-in, Wrappers in production AI systems

Related implementations:

Next: 03 · Expert — the production and systems concepts that turn this into something you can trust in front of real users.