01 · Foundations
The beginner concepts. Nothing else in the book makes sense until these do. Each section is: what it is, and why it matters. Where you've built something that relates to a concept, a Related implementations line links to your write-up in Learnings.
For one-line definitions of any term, keep the Glossary open alongside this.
Tokens
A token is the unit an LLM reads and writes — roughly ¾ of an English word. "Hello world" ≈ 2 tokens. The model doesn't see characters or words; it sees a sequence of token IDs and predicts the next one.
Why it matters: everything is priced and limited in tokens. Input tokens (your prompt) and output tokens (its response) both cost money, usually at different rates. When you reason about cost, latency, or "will this fit," you're reasoning about tokens. Output length is itself capped by a max-tokens budget — set it too low and a long response is truncated mid-generation, which can break anything parsing that output.
Context window
The context window is the model's working memory — the maximum tokens it can hold at once, counting input + output together. Modern models hold 100k–1M+ tokens.
Why it matters: the window is a hard wall. Everything the model "knows" for a given call must fit inside it: system prompt, instructions, retrieved documents, conversation history, and the room reserved for its answer. Managing what goes in that window deliberately is the whole discipline of context engineering.
Embeddings
An embedding is a list of numbers (a vector) that represents the meaning of a piece of text. Text with similar meaning produces vectors that sit close together in that number-space. Closeness is usually measured by cosine similarity (or, equivalently, cosine distance = 1 − similarity).
Why it matters: embeddings turn "find me things that mean roughly the same" into simple geometry. This is the engine under semantic search and therefore under RAG. You don't match keywords; you match meaning. What text you choose to embed determines what you can retrieve — embed the wrong field and retrieval quietly underperforms.
Temperature
Temperature controls randomness in generation. At 0.0 the model almost always
picks its single most-likely next token → deterministic, repeatable. Higher values
(0.7–1.0) let it sample less-likely tokens → more varied, more "creative," less
predictable.
Why it matters: for anything where you want the same structured answer every time — classification, JSON extraction, SQL — you want low temperature. For brainstorming or copy, higher. It's one of the cheapest knobs you have; a small bump is also a common trick to shake a model out of a repeated malformed response.
Model routing
Model routing means using a cheap/fast model for easy work and an expensive/smart model only for hard work — instead of paying frontier prices for everything.
Why it matters: most pipelines have a mix of "genuinely hard reasoning" steps and "just read this and classify it" steps. Sending both to the frontier model burns money and latency for no quality gain. Routing is often the single biggest cost lever you have. The judgment call is per step: is this reasoning, or is this reading?
Related implementations: Cheap model to classify, strong model to generate — routing a multi-phase pipeline by whether each step reasons or merely reads.
Why RAG exists
An LLM only knows what was in its training data. It doesn't know your data, and it will confidently make things up to fill the gap (hallucination). RAG (Retrieval-Augmented Generation) fixes this: before the model answers, you search a store for relevant documents and stuff them into the prompt, so the model answers from real source material instead of from memory. This is called grounding.
Why it matters: RAG is how AI systems "know" private, current, or domain-specific information without retraining the model. It's the default pattern for "make the LLM useful on our data."
The shape of every RAG system:
- Index (once, ahead of time): chunk your documents → embed each chunk → store the vectors.
- Retrieve (per query): embed the query → find the top-K most similar chunks.
- Generate: put those chunks in the prompt → the model answers grounded in them.
Tuning retrieval is a whole intermediate topic.
Structured outputs
A structured output forces the model to return data in a fixed, machine-readable shape (typically JSON matching a schema) instead of free prose. You then validate that shape before trusting it.
Why it matters: the moment an LLM's output feeds other code, prose is a liability — you'd be regex-parsing English. A schema turns the model into a reliable component: downstream code can rely on the fields existing and being the right type. This is one of the highest-leverage moves in applied AI. The one thing a schema doesn't decide is what to do when validation fails — that failure policy is a separate design choice (see graceful degradation).
Related implementations: Structured outputs, in practice — validating every pipeline phase into a typed object, and why the parse-failure path is its own feature.
Prompt anatomy
A prompt isn't one blob of text — it has parts, and where you put things matters.
| Part | Role |
|---|---|
| System prompt | Frames the model's role, rules, and constraints. Highest priority. This is what CLAUDE.md and stored prompt files are. |
| Instructions | The task: what to do, in what order, with what output format. |
| Context / reference | The material to work from — retrieved docs, examples, schema, domain glossary. |
| The input | The specific thing to act on right now (the ticket, the question). |
| Few-shot examples | 2–5 input→output pairs that show the model the pattern you want. |
Techniques worth knowing early: zero-shot (instructions only), few-shot (add examples — usually a big consistency win), chain-of-thought ("think step by step" before answering — helps on complex tasks).
What an agent actually is
Most "agents" aren't. The precise definition:
- Augmented LLM — an LLM with tools, retrieval, and/or memory. The basic building block.
- Workflow — a predefined sequence of LLM calls. The developer fixes the path; the model only fills in each step. Predictable, cheaper, easier to debug.
- Agent — an LLM in a loop with tools, where the model decides what to do next each turn (Think → Act → Observe) until it judges the task done. Flexible, but less predictable and harder to control.
The key distinction (Anthropic's): a workflow has the path fixed; an agent decides the path. Most real "agent" projects are workflows in disguise — and that's usually the right choice. Reach for an agent only when the task genuinely needs the model to choose its own steps. The mechanics of the workflow patterns (state machines, ReAct, tool use) are in the next tier.
Next: 02 · Intermediate — how these pieces get assembled and tuned into real systems.