Structured Outputs: How to Get Flawless JSON and XML from Any LLM

Quick Answer: Getting flawless JSON or XML from an LLM requires a multi-layered approach. For JSON, use the model's native Structured Outputs API mode by providing a JSON Schema (defined via tools like Pydantic models or Zod schemas). The provider validates outputs against the supplied schema, greatly improving reliability. For models that don't support native schemas, use XML tags as containment wrappers and prefill the model's response to start with the target bracket (e.g., { or <response>) to bypass conversational filler text.

What are native structured outputs?
Native structured outputs constrain an LLM's token generation using a pre-defined JSON Schema (such as a Zod or Pydantic schema) to ensure the output matches the schema, greatly improving reliability. Some providers validate the schema internally or may refuse/error instead of returning invalid JSON.

What is response prefilling in LLM APIs?
Response prefilling is an API technique (supported by some APIs, not all providers) where the developer pre-populates the assistant's response with a character like { to force the model to start outputting structured data directly.

When integrating Large Language Models (LLMs) into production software, the biggest source of failures is output parsing.

If your backend code expects a strict JSON object to update a database or trigger a webhook, a single misplaced comma, conversational intro ("Sure, here is the data you requested:"), or trailing backtick can crash your application.

To build robust, production-grade integrations, you must master the art of structured outputs and schema validation. Here is how to ensure your LLMs return flawless, machine-readable LLM JSON output and XML on every single request.

---

1. Native Structured Outputs: Schema Validation with Zod, Pydantic, and JSON Schema

How do native structured outputs work?

Native structured outputs constrain generation to follow the provided schema, preventing the model from producing invalid JSON structure.

If you are using modern models that support native structured outputs (such as recent OpenAI and Gemini models), the most reliable approach is to use the API provider's native Structured Outputs feature using a standard response_format configuration.

Instead of writing complex guidelines in your prompt asking the model to "be valid JSON," you supply a JSON Schema directly in the API request configuration. The generation process is constrained, preventing the model from outputting tokens that violate your schema.

Example using Zod (Next.js/TypeScript):

In Node.js, we define the schema using Zod and configure the response_format parameter in the OpenAI Responses API:

import OpenAI from 'openai';
import { zodResponseFormat } from 'openai/helpers/zod';
import { z } from 'zod';

const openai = new OpenAI();

const InvoiceSchema = z.object({
  invoiceNumber: z.string(),
  lineItems: z.array(z.object({
    description: z.string(),
    price: z.number(),
  })),
  total: z.number(),
});

// Configure standard response_format via the OpenAI Responses API
const response = await openai.chat.completions.create({
  model: "gpt-4o",
  messages: [{ role: "user", content: "Extract invoice data from the receipt..." }],
  response_format: zodResponseFormat(InvoiceSchema, "invoice"),
});

// Retrieve and parse the verified JSON response
const rawContent = response.choices[0].message.content;
const invoice = JSON.parse(rawContent || "{}");

Example using Pydantic (Python):

In Python, we define the schema using Pydantic models:

from openai import OpenAI
from pydantic import BaseModel, Field

client = OpenAI()

class LineItem(BaseModel):
    description: str
    price: float

class Invoice(BaseModel):
    invoice_number: str
    line_items: list[LineItem]
    total: float

completion = client.beta.chat.completions.parse(
    model="gpt-4o",
    messages=[
        {"role": "user", "content": "Extract invoice data from the receipt..."}
    ],
    response_format=Invoice,
)

invoice = completion.choices[0].message.parsed

Using native Structured Outputs ensures the model output matches your schema, greatly improving reliability. While most API providers will handle structured outputs by constraining generation or retrying internally, they may occasionally refuse the request or raise an error instead of returning invalid JSON.

For official documentation on setting up schemas, consult:

• OpenAI Structured Outputs Developer Guide
• Google Gemini API Structured Outputs Guide
• Anthropic Tool Use and JSON Mode Guide

---

2. Prompt-Based Structure: XML Delimiters

Why are XML Delimiters Useful for Prompt-Based Extraction?

XML delimiters are often easier for prompt-based extraction when native structured outputs aren't available, and they do not suffer from string-escaping bugs caused by double quotes. (For details on securing your inputs, read our guide on preventing prompt injection.)

If you are using models that do not support strict JSON schema constraints (or when using Claude for large reasoning tasks), you need to rely on prompt design. The most effective way to separate structured data in text-based workflows is using XML delimiters.

LLMs understand XML tags exceptionally well because they are trained on code and structured markup.

Why XML Delimiters Are Useful in Prompts:

1. Clear Boundaries: XML has explicit opening (<data>) and closing (</data>) tags, making it easy for a proper XML/HTML parser (such as BeautifulSoup or lxml in Python, or DOMParser in JavaScript) to extract the content.
2. Escaping Issues: JSON strings often break if the text content contains unescaped double quotes. XML tags contain content safely without character escaping bugs.

The Ideal XML Prompt Pattern:

Act as a user profile extractor. Extract data from the following text and wrap the outputs inside <profile> tags. Use the <name>, <age>, and <location> sub-tags.

<text>
John Doe is 34 years old and currently works out of New York City.
</text>

Your code can then parse the tags using a proper XML/HTML parser (rather than brittle regex) to extract the content without worrying about conversational drift.

---

3. Forcing JSON via Response Prefilling

How does response prefilling force valid JSON outputs?

Response prefilling supplies a starting character (like {) inside the assistant role message payload, forcing the model to generate the remaining JSON key-value pairs directly instead of introducing them with conversational filler.

If you must get JSON from a model without utilizing native schemas, the most common failure is the model's tendency to write chatty prefaces: "Here is the JSON object you requested:".

To bypass this, use Response Prefilling (supported by some APIs like Anthropic's Claude API, but not universally supported by all providers). In your API payload, you submit an assistant message containing just the opening curly brace {. The model is forced to continue typing directly from that brace, immediately outputting valid JSON fields.

API Payload Example:

[
  { "role": "user", "content": "Extract data into JSON." },
  { "role": "assistant", "content": "{" }
]

The model has no choice but to start writing keys immediately: "name": "John"... instead of talking first.

---

4. Comparing JSON Mode vs. Structured Outputs vs. Function Calling

When building applications with AI structured data, developers have three primary ways to enforce schema structure. Choosing the right one is essential:

Feature	JSON Mode	Structured Outputs	Function Calling (Tool Use)
Schema Enforcement	No (Guarantees valid JSON format, but not the specific schema fields)	Yes (Constrains generation to match the exact schema structure)	Yes (Constrains generation to match tool parameters)
Primary Use Case	General JSON generation without rigid constraints	Generating structured data, data extraction, direct UI rendering	Triggering external APIs, invoking tools, agentic workflows
Model Constraints	Requires prompt instructions to output JSON	Configured via API parameters (response_format)	Configured via API parameters (tools)
Error Handling	Susceptible to schema drift or missing keys	May error/refuse if schema is violated	May return invalid parameters; requires schema validation

---

Best Practices for Structured Outputs

• Handle Enums, Nullable, and Optional Fields: When defining your schemas in Zod or Pydantic, specify enum constraints for fields that have a fixed set of allowed values. Explicitly define optional fields or make them nullable if the value can be absent, preventing schema validation crashes.
• Handle Schema Rejections and Implement Retries: Always wrap your parsing code in a try/catch block. When a parser encounters an invalid format or validation fails (such as a missing required property or type mismatch), catch the exception, extract the validation error messages, and programmatically retry the request. Pass the error details back to the LLM in the next message, allowing the model to self-correct.
• Handle Streaming Structured Outputs: Streaming structured outputs requires special handling. Because the model outputs text sequentially, the stream will contain incomplete, invalid JSON chunks until the final token is generated. To parse streaming JSON in real-time, developers use partial-parsing libraries (like partial-json-parser) or SDK-provided streaming helpers that yield incremental updates of the parsed schema.
• Use Function/Tool Calling for External APIs: While structured outputs are ideal for extracting or formatting data to return directly to users, function calling (or tool calling) is preferable when invoking external APIs, since it signals to your orchestration layer that the model is ready to execute an action.
• Avoid Nested Complexity: Keep schemas as flat as possible. Highly nested arrays and objects increase the reasoning burden and cost more tokens.

---

Standardize Your Structured Output Templates

Building and maintaining correct JSON schemas and prompt layouts per model is a major engineering chore.

PromptBuff solves this by allowing developers to build, test, and save structured output templates. The platform lets you specify input parameters, target schemas, and model targets. It automatically formats the outgoing prompt with XML delimiters, injects prefilled assistant starters, or appends the correct JSON schema payloads to ensure your application code never breaks from parsing errors.

Take the headache out of JSON prompting. Sign up for PromptBuff.app today.

---

Continue reading: How to Use Prompt Caching to Cut Your LLM API Bills in Half · Preventing Prompt Injection: Securing Your AI App's System Prompts in Production