Structured Outputs Beat Prompt-and-Pray JSON Parsing

There's a line of code in countless production LLM systems — including the generation step of most RAG pipelines — that quietly causes 3 a.m. pages: json.loads(response_text). It works in the demo. It works for the first thousand requests. Then the model wraps its answer in a markdown fence, or adds a chatty "Sure! Here's the JSON:" preamble, or trails a comment, or hallucinates a field — and your parser throws, your pipeline halts, and you're writing a regex to strip backticks at midnight.

This is the prompt-and-pray pattern: you put "respond with valid JSON" in the prompt, cross your fingers, and parse the string. It is the wrong tool for a job that has a right tool. Modern providers offer structured outputs — schema-constrained generation that guarantees the model's output conforms to a JSON Schema you define. You stop hoping and start asserting.

Why prompt-and-pray fails

The failure isn't that models are bad at JSON. They're quite good. The problem is that "quite good" at scale means a small but nonzero failure rate, and that rate compounds. At 99% valid JSON, one in a hundred requests breaks — at a million requests a day, that's ten thousand failures.

The classic defenses are all bandages. Stripping markdown fences with string manipulation breaks on the next format the model invents. Wrapping json.loads in a retry loop doubles your latency and cost on the unhappy path and still fails if the model is consistently wrong. Few-shot examples of "good JSON" reduce the rate but never to zero. You are negotiating with a probabilistic system when you could be constraining it.

How structured outputs actually work

The mechanism is the difference. With constrained decoding, the provider restricts token sampling at each step to only tokens that keep the output valid against your schema. The model literally cannot emit a token that would produce invalid JSON or violate the schema's structure — closing braces, required fields, and types are enforced during generation, not checked afterward.

OpenAI exposes this as Structured Outputs with response_format and strict: true — see the OpenAI structured outputs guide. Anthropic supports reliable structured extraction through tool use, where you define a tool whose input schema is your desired output shape — see the Anthropic tool use docs. Either way, the contract moves from "please" to "guaranteed."

Define the schema once, in Pydantic

The cleanest workflow defines your output shape as a typed model and derives the JSON Schema from it — one source of truth for the constraint, the parsing, and your application's types.

from pydantic import BaseModel, Field
from typing import Literal
 
class Invoice(BaseModel):
    vendor: str
    amount_cents: int = Field(ge=0, description="Total in cents, never negative")
    currency: Literal["USD", "EUR", "GBP"]
    due_date: str = Field(description="ISO 8601 date, YYYY-MM-DD")
    line_items: list[str]
 
# OpenAI parses directly into your model:
from openai import OpenAI
client = OpenAI()
 
completion = client.beta.chat.completions.parse(
    model="gpt-4o-2024-08-06",
    messages=[
        {"role": "system", "content": "Extract invoice fields from the text."},
        {"role": "user", "content": raw_invoice_text},
    ],
    response_format=Invoice,   # schema-constrained generation
)
 
invoice: Invoice = completion.choices[0].message.parsed
print(invoice.amount_cents)   # typed, validated, no json.loads in sight

No string parsing. No fence stripping. The object you get back is already a validated Invoice with the right types. Your IDE autocompletes its fields, and a downstream type error surfaces at the boundary instead of three functions deep.

Schema design is the new prompt engineering

Constrained decoding guarantees the shape is valid. It does not guarantee the values are correct — the model can still extract the wrong number into a perfectly-typed int field. So your schema design carries real weight. A few principles that pay off:

• Use enums, not free strings, for any field with a fixed set of values. Literal["USD", "EUR", "GBP"] makes "US Dollars" impossible to emit.
• Add field descriptions — they're injected into the model's context and meaningfully guide extraction. Field(description="Total in cents, never negative") is doing prompt work.
• Model uncertainty explicitly. If a value might be absent, make the field Optional rather than forcing the model to invent one. Forcing a required field invites hallucination.
• Constrain ranges with ge/le where the domain allows it.

from typing import Optional
 
class ExtractionResult(BaseModel):
    found: bool = Field(description="True only if the value is present in the source")
    confidence: Literal["high", "medium", "low"]
    value: Optional[str] = Field(default=None, description="Null if found is false")

Giving the model a sanctioned way to say "I don't know" — a found: false path — is one of the highest-leverage schema decisions you can make. It converts hallucination into an honest null you can handle.

Validate values, even with a valid schema

Structural validity is necessary, not sufficient. Layer semantic validation on top of the schema for the things a JSON Schema can't express — cross-field rules, referential integrity, business constraints. Pydantic validators are the natural home for this:

from pydantic import model_validator
 
class Invoice(BaseModel):
    amount_cents: int
    line_items: list[str]
 
    @model_validator(mode="after")
    def non_empty_when_charged(self):
        if self.amount_cents > 0 and not self.line_items:
            raise ValueError("Charged invoice must have at least one line item")
        return self

Now an output that's structurally perfect but semantically nonsensical gets rejected at your boundary, with a clear error, instead of flowing downstream as silent bad data. This is the layered model: constrained decoding guarantees the shape, your validators guarantee the meaning.

Takeaways

• json.loads on raw model text is a latent production incident; stop praying and start constraining.
• Structured outputs use constrained decoding to make schema-conformant output a guarantee, not a hope.
• Define the schema once in Pydantic and derive constraint, parsing, and types from it.
• Schema design is prompt engineering: enums, field descriptions, and explicit optionality steer extraction.
• A valid shape isn't a correct value — layer semantic validators on top to catch nonsense the schema can't.