Skip to main content

LLM Workflow Patterns

Summary: A workflow is a fixed, developer-defined sequence of steps where an LLM fills in each step. Each pattern below is paired with how it's actually implemented in production.

At a glance​

QuestionAnswer
What is it?Reusable shapes for composing multiple LLM calls into one pipeline
Why does it matter?These aren't mandatory — they're a shared vocabulary; reach for one when your task's shape matches it
When do I use it?The task can be broken into fixed steps, even if each step's content is generated

Patterns and implementation​

Prompt chaining​

Sequential steps, output of one feeds the next, with a validation check between steps.

  • Example: generate an outline, validate it, then generate the full document from the validated outline.

Routing​

A classification step sends input down a specialized path.

  • Real-world implementation: keep classification as its own narrow call, returning structured output (see Structured Output With Tool Use) so downstream code branches on it reliably. Include a confidence signal with the classification; set a threshold, and route anything below it to a fallback path or human review instead of forcing a low-confidence guess. For genuinely ambiguous input, some teams allow multiple categories back instead of one, rather than forcing a single artificial choice.
  • Example: a customer service query classified as billing, technical, or general, each with its own tuned prompt; low-confidence queries fall back to general/human review. Anthropic also cites routing by difficulty — easy/common questions to a smaller model, hard/unusual ones to a more capable model — to optimize cost against performance.

Parallelization: sectioning​

Split one task into independent subtasks that run at the same time, then combine results. Subtasks are fixed by the developer in advance — this only applies when a task has genuinely independent internal parts.

  • Real-world implementation: in Python, this is typically plain asyncio plus the API's async client — fire off the independent calls with asyncio.gather rather than looping and waiting on each one sequentially, then merge the structured results in ordinary code (not another LLM call). No dedicated library is required unless the branches have complex dependencies on each other, at which point a framework like LangGraph starts to earn its keep.
  • Worth knowing: this is technically async concurrency (many calls in flight, overlapping while waiting on network I/O), not true multi-core parallelism. The field still calls it "parallelization" because the practical speed benefit is the same — the bottleneck is network wait time, not CPU.
  • Example: guardrails, where one model instance handles the response while a separate call screens the same input for inappropriate content — this tends to outperform one call trying to do both jobs at once.

Parallelization: voting​

Run the identical, complete task multiple times (not split) and aggregate the results for higher confidence.

  • Real-world implementation: run the same call N times, sometimes varying temperature or phrasing slightly, then take a majority result or only trust the output if it's consistent across runs.
  • Example: reviewing a piece of code for vulnerabilities with several different prompts, each flagging independently if it finds a problem, to balance false positives against false negatives.

Orchestrator-workers​

A central LLM dynamically breaks a task into subtasks, delegates each to worker LLMs, and synthesizes their results.

  • How it differs from sectioning: topologically similar, but the key difference is flexibility — sectioning's subtasks are fixed ahead of time by the developer; here the orchestrator decides the subtasks itself, at runtime, based on the specific input.
  • How it differs from an agent: it's still a workflow — the code path is fixed (orchestrator call → worker calls → synthesis call never changes shape between runs). Only the plan's content is dynamic, not the shape of the pipeline itself. See Agents for the full comparison table.
  • Real-world implementation: the orchestrator call typically returns a structured plan (a list of subtask descriptions), each subtask is dispatched as its own worker call, and a final synthesis call combines the worker outputs — three distinct LLM calls chained together, plus whatever concurrency you apply to the worker calls themselves.
  • Example: a coding tool that edits multiple files per task, where the number of files and the nature of each change can't be predicted in advance. Also common in search tasks that gather and analyze information from multiple sources.

Evaluator-optimizer​

One LLM performs the task, a second critiques and refines it.

  • Real-world implementation: keep the two roles as genuinely separate calls, not one call critiquing itself — a fresh call whose only job is to find flaws is more rigorous. Give the critic explicit criteria to check against (a checklist), and have it return structured feedback (see Structured Output With Tool Use) — e.g. a pass/fail field plus a list of specific issues — rather than free-form prose. Cap the loop at 2–3 revision rounds to avoid endless nitpicking. The revision call gets both the original output and the critic's specific feedback, so it knows exactly what to fix.
  • Example: literary translation, where an evaluator LLM catches nuances the translator missed on the first pass; or multi-round search tasks, where the evaluator decides whether another round of searching is warranted.

Build philosophy​

  • Default advice: keep it simple before reaching for a framework. Many production teams find frameworks add unneeded abstraction for straightforward chains.
  • Pick by category, not by name: decide which problem you actually have — structured output, orchestration, retrieval, observability — and only then choose a tool for it. Specific tool names age fast; the categories and the selection logic are the durable part, so re-evaluate the names periodically.
  • Common approach: composable functions in plain code — small, single-purpose functions (one for the LLM call, one for validation, one for extraction) chained together, rather than one large function doing everything at once. Easier to test, debug, and swap out individually.
  • Frameworks worth knowing:
FrameworkWhat it's forWhen to use vs. plain code
LangChainGeneral-purpose orchestration — chaining LLM calls, prompt templates, memory, broad third-party integrationsWhen you need many pre-built integrations quickly and don't mind an abstraction layer; step back to plain code once you need fine-grained control or hit debugging friction
LlamaIndexRetrieval-augmented generation — connecting an LLM to external documents/data, indexing, query enginesWhen the core job is retrieval over a non-trivial knowledge base; plain code is enough for a single simple vector search call
LangGraphGraph-based, stateful orchestration — explicit branching, loops, cyclical flows, shared stateWhen you need real branching or looping between steps (e.g. orchestrator-workers with retries, evaluator-optimizer's revision loop) that plain code would turn into tangled control flow
Plain codeComposable functions, each wrapping one LLM callThe default starting point — stay here until you hit genuine state/branching complexity a framework would meaningfully simplify

Remember​

  • Pick a pattern based on the task's shape: sequential dependency → chaining, distinct input types → routing, splittable independent subtasks known in advance → sectioning, confidence on one task → voting, subtasks unknown until runtime → orchestrator-workers, quality refinement → evaluator-optimizer.
  • Orchestrator-workers looks like sectioning but isn't — the difference is who decides the subtasks, and when. It looks like an agent but isn't — the difference is whether the pipeline's shape itself, not just its content, can change.
  • Start with plain composable functions; reach for a framework only when state or branching gets complex.

Sources​