Structured Outputs and Fallback Chains: How to Stop LLMs From Breaking Your Parser

Why structured outputs matter more than you think

The moment an LLM output feeds into another piece of code, you have a parsing problem. The model returns markdown when you asked for JSON. It hallucinates a field you never asked for. It truncates mid-object. It wraps the JSON in a friendly explanation. Every one of these failures becomes a 500 error in your application.

This is Part 3 of the AI Engineer Series. We have a harness from Part 1 and good retrieval from Part 2. Now we need outputs that are shaped the way our code expects, every single time. The problem is harder than it looks, and most teams underestimate it until it costs them a Saturday.

Three approaches, three different tradeoffs

You have three real options in 2026, and they are not interchangeable.

JSON mode tells the provider to constrain output to syntactically valid JSON. OpenAI and most providers offer it. It guarantees parseable JSON. It does not guarantee that the JSON matches your schema. The model can return {"answer": 42} when you wanted {"summary": "...", "confidence": 0.8}.

Tool calling (also called function calling) ties the model output to a typed schema. You define the shape, the model fills it. Anthropic and OpenAI both support this. The schema is enforced server-side, so the provider often produces valid output without retries. This is the strongest default for production work.

Constrained decoding masks tokens at sampling time so the model literally cannot emit invalid output. Libraries like Outlines and XGrammar implement this. It is the only approach that guarantees 100% schema validity, but you have to be self-hosting your model to use it.

The library that has settled the structured-output question for most Python teams is instructor. It works across 15+ providers with the same API, uses Pydantic models as the schema, and handles retries automatically when validation fails.

import instructor
from anthropic import Anthropic
from pydantic import BaseModel, Field
from typing import Literal

class SupportTicket(BaseModel):
    category: Literal["billing", "technical", "account", "other"]
    urgency: Literal["low", "medium", "high"]
    summary: str = Field(min_length=10, max_length=200)
    suggested_action: str

client = instructor.from_anthropic(Anthropic())

ticket = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=500,
    response_model=SupportTicket,
    messages=[{
        "role": "user",
        "content": "My card was charged twice for the $49 plan. Need refund."
    }],
)

print(ticket.category)  # "billing"
print(ticket.urgency)   # "high"

The failure modes you will actually see

Structured output failures are predictable. Plan for these six:

• Truncation: the model hit max_tokens mid-object. Always set max_tokens generously and check finish reasons explicitly.
• Hallucinated fields: the model returns keys you did not define. Strict Pydantic validation rejects these by default. Do not relax it.
• Type drift: your schema says int, the model returns "42" as a string. Pydantic will coerce by default. Decide whether you want coercion or strict types per field.
• Enum invention: you specified Literal["billing", "technical", "account", "other"], the model returns "refund". Validation catches it, but you need the retry to actually fix it.
• Null evasion: the model returns null or {} when it does not know, rather than the structured "I do not know" you defined. Add an explicit "uncertain" enum value or an Optional reason field.
• Wrapper noise: the model returns valid JSON inside markdown backticks or a friendly preamble. Most libraries strip this automatically, but raw API calls do not.

The fallback chain pattern

The right model for structured outputs is not "the strongest one." It is "the cheapest one that gets it right, with an escalation path when it does not." This is the canonical use case for graceful degradation:

from pydantic import ValidationError

def extract_with_fallback(text: str, schema):
    """Try cheap model first, escalate on failure."""
    
    # Tier 1: cheap and fast
    try:
        return client.messages.create(
            model="claude-haiku-4-5",
            max_tokens=500,
            response_model=schema,
            max_retries=2,  # instructor retries internally
            messages=[{"role": "user", "content": text}],
        )
    except (ValidationError, instructor.InstructorRetryException):
        pass
    
    # Tier 2: stronger model, same schema
    try:
        return client.messages.create(
            model="claude-sonnet-4-6",
            max_tokens=1000,
            response_model=schema,
            max_retries=2,
            messages=[{"role": "user", "content": text}],
        )
    except (ValidationError, instructor.InstructorRetryException):
        pass
    
    # Tier 3: degrade schema and try again
    return client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=1000,
        response_model=schema.relaxed(),  # optional fields, looser types
        messages=[{"role": "user", "content": text}],
    )

Most production calls return on Tier 1. The few that fail escalate quietly. The Tier 3 fallback exists for the long tail. The whole chain is observed at the call site so you can see how often each tier fires.

Validators that improve quality

Pydantic validators run before instructor accepts an output. If a validator raises ValueError, instructor catches it, feeds the error back to the model, and retries with that feedback in the prompt. This means you can encode domain constraints directly:

from pydantic import BaseModel, field_validator
from datetime import datetime, timezone

class Invoice(BaseModel):
    total_usd: float
    invoice_date: datetime
    line_items: list[str]
    
    @field_validator("total_usd")
    @classmethod
    def positive_total(cls, v):
        if v <= 0:
            raise ValueError("total_usd must be positive")
        return round(v, 2)
    
    @field_validator("invoice_date")
    @classmethod
    def not_future(cls, v):
        if v > datetime.now(timezone.utc):
            raise ValueError("invoice_date cannot be in the future")
        return v
    
    @field_validator("line_items")
    @classmethod
    def has_items(cls, v):
        if not v:
            raise ValueError("invoice must have at least one line item")
        return v

Now when the model returns a future date or a negative total, the validator rejects it, and the next attempt has the error message in context. The model corrects itself most of the time.

The single rule that catches the most bugs

Never silently default missing fields. If the model did not return a required field, that is a signal worth logging. Many teams write code like:

# BAD
confidence = result.get("confidence", 0.5)

This is how quality regressions hide for months. The field is missing because the model decided not to return it. You just replaced its decision with a constant. Catch the missing field, log it, and treat it as a data point. Pydantic with required fields does this by default. Resist the urge to make everything Optional.

When to use what

A practical decision tree:

• Working with Claude, GPT, or Gemini, want the cleanest path: tool calling via instructor
• Need rough JSON shape but not strict schema: JSON mode, no library
• Self-hosting an open model and need 100% schema validity: constrained decoding with Outlines or XGrammar
• Working with smaller open models where structured output is unreliable: constrained decoding wins, no contest

For agent workflows that go beyond single-shot extraction, look at PydanticAI. It is built on the same Pydantic foundation, adds typed tools and dataset replays, and is the natural next step when your structured-output code starts growing into an agent loop. We will come back to agents in Part 12.

What to build this week

Extend your harness with structured outputs and a fallback chain:

• Pick a real extraction task (invoice parsing, support ticket classification, resume extraction)
• Define a Pydantic model with at least one validator
• Wire instructor into your harness
• Build a 3-tier fallback: Haiku, Sonnet, Sonnet with relaxed schema
• Deliberately break it: pass garbage input, pass an input that triggers an enum violation, pass an input where the answer truly is "I do not know"
• Observe how each tier fires

You will likely find that Tier 1 handles 95% of well-formed inputs and Tier 2 handles most of the rest. That ratio is what makes structured outputs viable at scale.

What is next

Part 4: evals. Without measurement, every optimization decision becomes a guess. We will look at deterministic assertions, LLM-as-judge done right (with bias mitigation), pairwise versus pointwise scoring, and the eval set discipline that separates teams who ship from teams who hope.

Previous in the series

• Part 1: Harness Engineering
• Part 2: Context Engineering & Retrieval Quality

References

1. instructor documentation by Jason Liu. The de facto standard for structured outputs in Python, with 3M+ monthly downloads.
2. Anthropic Tool Use documentation. The native approach to structured outputs on the Claude API.
3. OpenAI Structured Outputs guide. The Responses API approach for OpenAI models.
4. Outlines documentation. Constrained decoding for self-hosted models with strict grammar guarantees.
5. XGrammar. Fast constrained decoding implementation, integrated into vLLM and SGLang.
6. Pydantic field validators. The pattern that lets you encode domain constraints directly into your schema.