Lesson 27: Taming the Output — Forcing LLMs to Return Structured JSON
What We’re Building Today
A four-layer JSON extraction pipeline: JSON-mode prompt → Zod-equivalent schema validation → retry-with-correction → regex fallback extractor
A 10,000-classification stability harness that measures recovery rate at each layer and exits nonzero if total valid JSON falls below 99.9%
A structured output engine integrated into NEXUS so every downstream consumer—vector indexer, moderation router, Qdrant payload builder—receives typed, validated classification records
Why This Matters
In 2022, Slack’s ML Platform team published a retrospective on their Semantic Search rollout. The system used GPT-3 to classify message intent before routing to downstream indexers. Under load, the model began emitting partial JSON—valid up to the closing brace of the first key, then truncating mid-string. The downstream parser threw, the indexer silently dropped the message, and 4.3% of search entries went missing over a 72-hour window before anyone noticed. The failure mode wasn’t hallucination. It was structural: the model’s output was almost valid JSON, which is worse than completely invalid JSON because it bypasses length-based sanity checks.
NEXUS faces the same risk. Day 26 shipped a toxicity detector that returns labels and confidence scores. Day 25 built the Redpanda consumer pipeline that routes those results. Neither layer can tolerate a malformed payload. Without structured output enforcement, one pathological prompt—long content, unusual Unicode, nested quotes—brings down the classification pipeline.
Core Concepts
JSON Mode Is a Hint, Not a Contract
Ollama’s format: "json" parameter instructs the model to bias its sampling toward JSON-compatible tokens. The mechanism is a constrained beam search: at each decoding step, the sampler suppresses tokens that would make the partial output unparseable as JSON. This works reliably for short outputs with flat schemas. It breaks down for deeply nested structures, long string values, or when the model’s internal “end of thought” signal fires before the JSON is structurally complete—producing {"label": "toxic", "score": 0. with no closing brace.
The production tradeoff: JSON mode reduces malformed output from ~8% to ~0.3% at P99. It does not eliminate it. You still need validation downstream.
