Choosing Models for Structured Output
Selecting a language model for applications that require fixed, predictable output requires a different perspective than choosing a model for general creative or narrative tasks. In practice, it turns out that a language model's general intelligence is barely correlated with the reliability with which it adheres to a predefined data schema. A model can reason excellently about complex questions in terms of content, yet at the same time structurally fail to exactly maintain point-by-point data structures. Anyone building applications that automatically process data must therefore consciously consider the unique properties needed for structured output.
Choosing the right model directly affects the architecture of your systems. When you're looking for the right balance between model size, speed, and reliability, it's worthwhile to look at broad categorizations such as those described in choosing a model per task, where the underlying skills are weighed per use case. Structured output, however, requires specific guarantees that go deeper than a simple instruction in the prompt.
Why Structured Output Is a Separate Model Property
Many users wrongly assume that language models naturally understand what a data structure is. Based on their training history, however, models are primarily trained to predict the next token in a continuous stream of free text. Imposing a strict schema means the model must constantly check during generation whether the chosen tokens meet syntactic and semantic constraints.
This ability is closely tied to the internal representation of grammar and syntax within the model's weights. Models that score poorly here often lose the structure halfway through a longer generation, forget required closing brackets, or add unsolicited text outside the expected structure. The skill of following instructions meticulously is often tested via standardized evaluations; an in-depth analysis of this can be found in the analysis of instruction-following with the IFEval standard, which provides insight into how models handle rigid formatting requirements.
The Three Levels of Provider Guarantees
When comparing model architectures in the area of structured output, providers generally offer three different levels of support. It is crucial to understand what these levels actually guarantee before making a choice for your production environment.
| Level | Functionality | What it actually guarantees |
|---|---|---|
| 1. Free text with instruction | The model is simply asked, via the prompt, to produce JSON, for example. | No guarantee whatsoever. The model can add text before or after the object, make syntax errors, or forget fields. |
| 2. JSON mode | An API setting that enforces that the output starts with a curly brace and ends with a curly brace. | Valid syntax (syntactic correctness), but it does not guarantee that your specific schema, field names, or data types are followed. |
| 3. Schema-constrained decoding | The model is provided with a formal schema (such as JSON Schema) that is enforced during generation. | That every generated token strictly complies with the specified schema and the allowed types, making syntax errors impossible. |
The third level changes the fundamental operation of the generation process. Instead of blindly trusting the probability of the next letter, the underlying inference layer intervenes at the logit level. Tokens that do not match the active schema are assigned a probability of zero and are immediately eliminated.
The Cost of Schema-Constrained Decoding
Although enforcing a schema at the token level drastically reduces the error rate, it also brings concrete disadvantages and limitations that you must weigh in your model selection.
First, narrowing the search space sometimes leads to unforeseen restrictions in the model's flexibility. If the schema is extremely complex or deeply nested, the inference engine can struggle to efficiently navigate the allowed paths. This results in slower generation time per token. In addition, not all open-source or commercial models support every conceivable feature of standards such as JSON Schema. Complex constructs with conditional dependencies (such as anyOf or oneOf with intricate restrictions) are simply ignored by smaller models or lead to API errors.
Second, models lose part of their inherent 'thinking space'. Because every generated character must directly conform to the straitjacket of the schema, the model is less able to incorporate intermediate reasoning into the output, unless the schema explicitly reserves a separate field for this.
Where Things Go Wrong in Practice, Even with Valid JSON
A common misconception is that schema-constrained decoding solves all errors. Even when a model produces a syntactically one hundred percent valid JSON structure that neatly complies with the schema, problems still regularly arise at the semantic level in practice. When selecting a model, you should watch for the following risks:
- Fabricated field values: The model fills in fields with plausible-sounding information that is factually incorrect or was not present in the source text (hallucination within the structure).
- Enumerations out of range: If a schema prescribes a fixed list of allowed values (enums), weaker models can still invent creative variants if the prompt is too complex.
- Numbers as text: Models sometimes deliver numeric values as strings (for example
"125.50"instead of125.50), which causes errors in downstream databases. - Dates in varying notation: Without strict instructions, a model easily switches between ISO notations, American dates, and readable text strings within the same field.
Note: Syntactic validation is only the first line of defense. A model that performs well on structured tasks must have sufficient semantic understanding to consistently apply data types and constraints across longer documents.
Schema Design as the Biggest Quality Lever
Before you decide to switch to a larger and more expensive language model, it's important to recognize that the design of your schema often has a greater influence on reliability than the model itself. Simple adjustments to the data structure can enormously improve the accuracy of weaker models.
In your design, always prefer flat structures over deeply nested objects. The fewer hierarchical levels the model has to hold in working memory at once, the smaller the chance of forgotten closing structures. Make enumerations explicit and keep the field descriptions in the schema clear and factual.
A proven design pattern is including a specific field for missing information, such as a boolean or a string with the value "niet gevonden". If you don't include this field element, you force the model to guess when data is missing from the source text just to fill the required schema anyway. By offering an explicit way out, you give the model room to honestly decline to provide the requested data.
Function Calling as a Variant of the Same Problem
Function calling (or tool use) is at its core nothing more than a specialized form of structured output. Instead of a generic data schema, the programmer provides a list of available functions and corresponding arguments, after which the model decides which function to call and with which parameters.
Models that perform well at function calling generally also show superior reliability in regular schema-constrained output. They are trained to translate intentions directly into formal argument lists. Anyone who wants to set up complex interactions in which models autonomously perform actions should therefore look at the criteria that apply to model selection for agent systems; you can read more about the broader implications of this in the guide on selecting models for agents.
What You Measure to Choose the Right Model
Choosing a model based on marketing claims or theoretical specifications rarely leads to the best results in a specific production environment. You will need to set up your own representative set of test cases and apply the following metrics:
- Validation success without recovery: The percentage of generations that pass validation by the local validator on the first try, without needing a programmatic recovery loop (retry) or correction prompt.
- Behavior with missing information: How does the model respond to ambiguous input or documents in which the requested data is simply missing? Does it validate cleanly against the expected empty values, or does it go off the rails?
- Stability with long input: Does the schema's accuracy hold up as the input text grows from a few paragraphs to dozens of pages of documentation?
The Practical Trade-off: Large versus Small
A persistent misconception is that you always need the largest and most powerful model on the market as soon as you work with structured data. In practice, this is more nuanced. A smaller model with built-in schema-constrained decoding often performs better in terms of syntax and field fidelity than a huge model that can only be steered via free-form instructions.
The economic and operational advantages of smaller models are significant: they generate faster, consume fewer resources, and have lower API costs. If you limit the complexity of your schema and choose a model that natively handles structured decoding, a compact architecture can suffice for the vast majority of your routine data processing tasks.
