JSON mode is reliable at one thing: when its prerequisites are met, the model returns syntactically valid JSON. It is not reliable at the thing most people assume it does, which is returning the fields, types and values your code expects. For that you need schema-constrained output, plus validation on your side. The rest of this article explains where the line sits and how to build around it.
Key takeaways
- JSON mode targets valid syntax only. It does not enforce your schema, so keys can be missing, renamed or the wrong type.
- Truncation is the quiet failure: a response cut off at the token limit is not complete JSON, so check the finish reason before you parse.
- Schema-constrained output (structured outputs, strict tool use) closes the structural gap, but not the semantic one. Valid JSON can still be wrong.
- A production pipeline needs local validation, a capped retry loop, and logs of raw and parsed output.
- Test the ugly paths before launch: truncation, refusals, unsupported schema features and the cost of retries.
Table of Contents
- What JSON mode actually guarantees
- JSON mode vs structured outputs
- The four layers where JSON output fails
- Truncation: the failure that looks like success
- Refusals and other non-normal completions
- Validation that survives production
- A pre-launch test checklist
- Perspective: boring validation beats clever prompts
- Getting this built into your product
- FAQ
- Sources
What JSON mode actually guarantees
OpenAI's own documentation says JSON mode ensures valid JSON when the supported model, message and tool prerequisites are met, except in edge cases. Two things in that sentence matter.
First, "except in edge cases". The same page says applications must handle cases where the output is incomplete. So even the syntax guarantee has a hole, and I come back to it in the truncation section.
Second, nothing there mentions your schema. JSON mode stops the model from wrapping output in prose or code fences, and that is a real improvement over prompting alone. But it will not stop a model from returning {"total": "about forty"} when your code needs a number called total_cents.
There is also a setup trap. With OpenAI, at least one message in the input must contain the word "json". Without it, the model may emit whitespace until it hits the token limit. I have seen this exact bug cost a team an afternoon, because the API call "succeeded" and returned nothing useful.
What JSON mode does not guarantee
- That required keys are present.
- That values have the right type or fall inside an allowed range.
- That an enum field contains one of your allowed values.
- That the content is true, or consistent with your business rules.
- That the response is finished.
If your downstream code reads result.invoice.total, any of those failures is a production incident, not a parse error you can catch cleanly.
JSON mode vs structured outputs
The useful mental model is three levels of control, each stronger than the last.
| Approach | What it gives you | What is still on you |
|---|---|---|
| Prompting for JSON | Often works, sometimes doesn't | Everything: parsing, repair, schema, retries |
| JSON mode | Syntactically valid JSON in normal completions | Schema, types, completeness, semantics |
| Schema-constrained output | Output limited to your supplied schema | Semantics, refusals, unsupported features, truncation |
Structured Outputs from OpenAI uses constrained decoding: generation is limited to tokens that are valid under the schema you supply. That is a different mechanism from JSON mode, which is why it can promise much more. OpenAI also notes that Structured Outputs works with newer models such as gpt-4o-2024-08-06 and later, while JSON mode remains the option on older ones. So model choice can decide which tool you are allowed to use.
Other providers reach the same goal differently. Anthropic documents structured outputs through constrained decoding, with a JSON output format and a strict mode for tool inputs. The vocabulary changes between vendors; the question to ask does not. Is the schema enforced during generation, or merely requested in the prompt?
When the provider enforces it, schema output is the more reliable path. When it doesn't, you are back to prompting plus validation, which is significantly less reliable than native features.
When JSON mode is still the right choice
Use it when your model doesn't support schema enforcement, or when the output shape really is open-ended and you only need something parseable. For anything that feeds typed code, a payment, a database write or an email to a customer, start with schema-constrained output and treat JSON mode as the fallback.
The four layers where JSON output fails
It helps to stop thinking of "JSON reliability" as one number. A pipeline has several failure layers, and each needs its own handling:
- Transport. Timeouts, rate limits, 5xx errors. Nothing to do with JSON, but they arrive in the same code path. If you run volume, read how to check, handle and raise OpenAI rate limits before launch.
- Completion. The model stopped for a reason other than finishing: token limit, refusal, content filter.
- Structure. Broken syntax, or valid syntax that violates the schema.
- Semantics. Valid, schema-conformant JSON that is wrong for your business: an invoice date in the future, a discount of 140%, a customer ID that doesn't exist.
A 2026 empirical study of structured output failures makes a point worth remembering: aggregate validity rates can hide distinct failure patterns, so classifying failures helps you choose the right remedy. A single "98% valid" number tells you little. Is the other 2% truncation, bad enums or refusals? The fix for each is different.
Truncation: the failure that looks like success
If the model hits your max_tokens limit mid-object, you get half a document. JSON mode cannot rescue that, because the stream simply ended. OpenAI says applications must handle incomplete-object edge cases.
The habit that prevents it is simple: read the finish reason on every response before you parse. If it says the output stopped because of length, don't try to repair the JSON. Either retry with a higher limit, or split the task so each response is smaller.
A practical rule I use: estimate the largest realistic output, then set the limit comfortably above it. Long arrays are the usual culprit, such as extracting every line item from a 40-page contract into one response. Ask for pages or sections separately and merge in code.
Refusals and other non-normal completions
A model can decline a request. With free-form JSON, a refusal may arrive as a polite paragraph that breaks your parser, or worse, as a JSON object with empty fields that passes your parser.
OpenAI states that Structured Outputs makes safety-based refusals programmatically detectable. That is a real operational benefit: you can route a refusal to a human instead of writing it to your database as data.
One more constraint to know: Structured Outputs is not compatible with parallel function calls, and enabling them can produce outputs that don't match your schema. If your agent flow calls several tools at once, check this setting before you trust the guarantee.
Validation that survives production
Even with strict schema enforcement, you need a validation layer in your own code. JSON Schema describes shape. It cannot express "the end date must be after the start date" or "this SKU must exist in our catalog". One practitioner puts it plainly: semantic validation is beyond JSON Schema's scope, so a local layer stays necessary.
In a TypeScript stack this is usually Zod; in Python, Pydantic. The pattern is the same:
- Parse the string. If parsing fails, record the failure and go to the retry path.
- Validate against a typed schema in your code, the same one you sent to the model where possible.
- Run business rules: ranges, cross-field checks, lookups against your database.
- Only then write to storage or call the next system.
A clean example is a Zod schema for an extracted invoice:
const Invoice = z.object({
vendor: z.string().min(1),
total_cents: z.number().int().nonnegative(),
currency: z.enum(["USD", "EUR", "GBP"]),
issued_on: z.string().date(),
}).refine(v => new Date(v.issued_on) <= new Date(), {
message: "issued_on is in the future",
});
The refine line is the semantic layer. No schema constraint at generation time would have caught that.
Retries: useful, but cap them
When validation fails, retrying with the validator's error message often fixes it. The same empirical study notes that a repair attempt can include concise validator errors, but the retry loop should be capped. I use one or two retries, then route the case to a human queue. An uncapped loop turns a rare failure into a cost spike and a latency problem.
MLflow's guidance on production structured output agrees with the general shape: validate locally, retry with a tightened prompt or escalate to review, and log both raw and parsed output. The logging part matters more than it sounds. When a customer says "your system got my invoice wrong", the raw model output is the only evidence you will have.
A pre-launch test checklist
Before you rely on JSON output in a workflow someone pays for, run these tests deliberately rather than waiting for users to find them:
- Truncation. Force a tiny token limit and confirm the code detects it and doesn't parse the fragment.
- Refusal. Send a request the model should decline and confirm it lands in your review path.
- Schema subset. Providers support only part of JSON Schema. Confirm each of your schemas is accepted, including nested objects and enums.
- Semantic edge cases. Empty documents, documents in the wrong language, duplicate entries, absurd numbers.
- Retry economics. Measure how often retries fire and what they add to latency and cost on your real inputs.
- Model upgrades. Re-run the same set whenever you change model versions. Regressions arrive silently.
That last point connects to evaluation. A fixed set of test cases that you re-run on every change is the cheapest way to catch drift, and I wrote up how to build one in AI evaluation metrics for production LLMs. If the problem you are really chasing is wrong answers rather than broken syntax, reducing hallucinations in production goes into what retrieval alone does and does not fix.
The model choice also changes how much of this you carry. Hosted models with native schema enforcement give you more guarantees out of the box; self-hosted ones may need grammar-constrained decoding you configure yourself. I compare that trade-off in open source vs OpenAI for your product.
Perspective: boring validation beats clever prompts
Most JSON problems I get called in to fix are not model problems. They are missing-plumbing problems. A founder ships a feature that works in the demo, because the demo input is clean, and then production sends a 60-page PDF, a blank upload and a document in Polish, all in one week.
My view is that the prompt is the least important part of reliable JSON. What matters is that every model response is treated as untrusted input, exactly like a form submission from the internet. You would never write a user's form field straight to your database; don't write a model's output there either.
So I build the same boring things every time: schema-constrained output where the provider supports it, a typed validator in the app, a business-rule layer, a capped retry, a human review queue for what's left, and logs of raw output. None of it is clever. That is the point. The next developer who opens the code should recognize every piece, and that is also why I keep the stack to widely known tools like TypeScript and Node.
One honest caveat: if your use case is a one-off internal script, you don't need all of this. Use JSON mode, check the finish reason, and move on. The full layer cake is for output that touches money, customers or compliance.
— Hanad Kubat
Getting this built into your product
If you are adding an AI feature to a SaaS product, or you have a prototype whose LLM calls break on real data, the pipeline above is a standard part of what I build. I work alone, at a fixed price: builds start at €12,000 and the client owns the code from the first commit. You can see how an engagement works on my main page, or start with a fixed-price audit of what you have.
FAQ
Is JSON mode 100% reliable?
No. It aims to guarantee syntactically valid JSON when prerequisites are met, with documented edge cases such as incomplete output. It doesn't guarantee your schema, and it can't make the content correct. Treat it as one layer, not the whole solution.
What is the difference between JSON mode and structured outputs?
JSON mode targets valid syntax. Structured outputs constrain generation to a schema you supply, so required keys and types come back as defined. Structured outputs is the better default wherever the model supports it.
Why does my JSON mode response come back empty or full of whitespace?
With OpenAI, the input must explicitly mention JSON, and the model may otherwise emit whitespace until the token limit. Add a clear instruction such as "Return a JSON object with these fields" to the system or user message, and check the finish reason.
Do I still need Zod or Pydantic if I use structured outputs?
Yes. Schema enforcement covers shape, not meaning. A local validator catches regressions and business-rule violations, such as a date in the future or an ID that doesn't exist in your system.
How many retries should I allow?
One or two is a sensible starting point, with the validator's error message included in the retry. Beyond that, send the case to human review. Measure how often retries fire, since they add both latency and cost.
Sources
- Function Calling in the OpenAI API, OpenAI Help Center
- Introducing Structured Outputs in the API, OpenAI
- Structured model outputs, OpenAI
- Structured outputs, Claude Platform Docs
- Why Does Structured Output Fail? An Empirical Study
- Enforce, Validate, Observe: 3 LLM Structured Output patterns, MLflow
- Getting JSON You Can Trust Out of an LLM, Adam Innes
- Structured Outputs Are Not a Solved Problem