Skip to main content

Validating every phase into a schema made LLM output safe to build on

Summary: Each phase returns a validated typed object, never prose, so downstream code can rely on the fields existing and being the right type. The one thing a schema doesn't decide โ€” what to do when validation fails โ€” is a separate, deliberate choice.

What was encounteredโ€‹

The model's output isn't the end of the line โ€” it's the input to the next function. The moment clarify's output feeds db.append_clarify_result() and gets posted to Shortcut, prose is useless: I need result.extract_sql to exist and be a string. Free-text output would mean regex-parsing English.

How it was solvedโ€‹

Every phase parses into a Pydantic model โ€” ClarifyResult, SessionContext, DraftResult. The parse is two steps: strip any ```json fences, then Model.model_validate_json(). That validation is the trust boundary โ€” once it passes, the rest of the code treats the object as reliable and never re-checks it.

Reference: Structured outputs โ€” a concrete instance of it. The non-obvious follow-on: a schema gives a clean failure signal but not a failure policy; deciding what happens on a failed parse is its own design choice, and getting it wrong bit me โ€” see the silent JSON-parse failure.