Skip to main content

LLM Workflow App Practices

Summary: Once the pipeline shape is fixed, the remaining work is operational — structured output at every boundary, versioned prompts, retries, per-step tracing, and evals that gate prompt changes the way tests gate code.

At a glance​

QuestionAnswer
What is it?A set of practices for building LLM apps with fixed, code-driven pipelines
Why does it matter?Prevents most of the "plumbing" pain (parsing, retries, debugging) that shows up in a first LLM app, and keeps it trustworthy as prompts and models change
When do I use it?Building any app where the LLM is called at known steps, not deciding its own next action

How it works​

Architecture

  • Separate orchestration (which step runs next) from prompt logic (what each step asks the model)
  • One LLM call = one clear responsibility
  • Keep control flow deterministic — the code decides the next step
  • Make each step idempotent so retries don't cause side effects (e.g. double API charges)

Prompts

  • Store prompts as versioned files/templates, not inline strings
  • Separate system/instruction prompt from the user content being processed
  • Keep each prompt narrow — one task per prompt
  • Treat prompts as deployable artifacts, not code strings, so a bad prompt can be rolled back without a full redeploy

Output handling

  • Request structured output (JSON / tool-calling) instead of parsing free text
  • Validate schema, types, and ranges at the boundary before passing downstream
  • Decide upfront: on invalid output, retry, fallback, or fail the whole run?

Reliability

  • Generic retry-with-backoff wrapper, not per-call copy-paste
  • Set timeouts; handle rate limits
  • Use a circuit breaker so one failing step doesn't keep hammering the next
  • Pin model versions explicitly — providers deprecate and swap models, and an unpinned app can silently change behavior

Observability

  • Log the actual prompt sent and response received per step, not just pass/fail — a tracing tool (LangSmith, Langfuse) does this for near-zero effort, so adopt one early rather than hand-rolling it
  • Track token usage and latency per step; treat a latency regression (e.g. p95 doubling) as a quality issue, not just a performance one
  • Set a per-call token budget alert to catch cost blowups from malformed input
  • Keep the ability to replay a failed run with the same input

Evals (distinct from logging)

  • Logging shows what happened; evals show whether it was good
  • Build a hand-written eval set from real input/output samples before picking any eval tooling — the dataset matters more than the framework
  • Run offline evals in CI that block a merge if a prompt change regresses quality
  • Run online evals against live production traces to catch drift, not just test-case performance

Security & data handling

  • Sanitize/validate user input before it enters a prompt to reduce prompt-injection risk
  • Watch for PII appearing in outputs or logs — the model can echo sensitive input downstream
  • Know what data leaves the system — anything sent to a third-party LLM provider is a data-handling decision, not just an engineering one

Cost & performance

  • Use a cheaper model for simple steps (e.g. classification), stronger model for generation
  • Cache identical/near-identical requests
  • Shorter context is cheaper and faster, but don't starve the model of needed info
  • Outputs aren't fully reproducible even at temperature 0 — inference kernels aren't batch-invariant, so results shift with whatever else happened to be batched alongside your request. Don't build on the assumption that identical input gives identical output

Example​

A step calls a paid enrichment API and writes the result to the database. The write fails on a transient connection error, so the generic retry wrapper runs the step again — and the enrichment call is billed a second time. Making the step idempotent (do the paid call once, store the result under a key, and let the retry resume from the stored value) is what separates a safe retry from a duplicated side effect.

Remember​

  • Structured output + validation at the boundary removes most "plumbing" pain — usually the highest-leverage first fix.
  • Evals are to prompts what tests are to code — shipping a prompt change without an eval is shipping untested code.
  • Skip agentic frameworks and multi-agent orchestration for a fixed pipeline — they solve a different problem.
  • Filter new articles by durability: advice that would apply to any unreliable third-party API (retries, logging, validation) is durable; advice that only makes sense because "the model might act autonomously" is agent-specific.

Unverified​

Not confirmed against a primary source — these read as sound, but treat them as reasoning rather than established practice:

  • Rotating part of the eval set periodically, on the theory that cases become trivial once prompts have been tuned against them.
  • Tracking refusal-rate drift after a model upgrade, on the theory that a new model version can quietly make the app over-refuse.

Sources​