Tutorial 4: Structured Outputs (JSON and Beyond)

Free-form text outputs are easy to read; hard to parse. Structured outputs (JSON, XML, etc.) let your code consume LLM responses reliably. This tutorial walks through getting them.

What You'll Build

A prompt that returns reliable JSON your code can parse without defensive cleanup.

Step 1: Decide on the Schema (10 min)

Define what you want back:

interface TicketSummary {
  issue: string;
  category: 'billing' | 'technical' | 'account' | 'other';
  urgency: 'low' | 'medium' | 'high';
  action_required: boolean;
  next_step: string | null;
}

Specific types. Constrained values where possible.

Step 2: Write the Prompt (10 min)

You are a support ticket analyst. Output your analysis as JSON 
matching this schema:

{
  "issue": "Brief description of the customer's issue",
  "category": "billing" | "technical" | "account" | "other",
  "urgency": "low" | "medium" | "high",
  "action_required": boolean,
  "next_step": "What support should do next, or null if no action"
}

Output only valid JSON. No preamble, no explanation, no markdown 
formatting.

Ticket to analyze:
{ticket_text}

Explicit schema. Explicit format.

Step 3: Use the Native JSON Mode (varies)

Many APIs have a native JSON mode:

OpenAI:

response = client.chat.completions.create(
    model="gpt-4",
    response_format={"type": "json_object"},
    messages=[...]
)

Anthropic:

Mention "JSON only" in the prompt; the model produces JSON. Or use tool use (function calling) for guaranteed structure.

Native modes are more reliable than text-based prompts.

Step 4: Parse and Validate (15 min)

import json
from pydantic import BaseModel
from typing import Literal, Optional

class TicketSummary(BaseModel):
    issue: str
    category: Literal['billing', 'technical', 'account', 'other']
    urgency: Literal['low', 'medium', 'high']
    action_required: bool
    next_step: Optional[str]

def parse_response(text):
    try:
        data = json.loads(text)
        return TicketSummary(**data)
    except (json.JSONDecodeError, ValueError) as e:
        # Handle invalid response
        return None

Pydantic (or similar) validates the structure. Type errors are caught immediately.

Step 5: Handle Failures Gracefully (15 min)

Models occasionally produce invalid JSON. Handle it:

def get_summary_with_retry(ticket_text, max_retries=2):
    for attempt in range(max_retries + 1):
        response = call_llm(ticket_text)
        result = parse_response(response)
        if result:
            return result
        # On failure, try again (different sampling)
    
    # All attempts failed; log and return fallback
    log_failure(ticket_text, response)
    return None

Don't crash on parsing failure. Retry; fall back.

Step 6: Use Tool/Function Calling (advanced, 20 min)

For guaranteed structure, use function calling:

# Anthropic example
tools = [{
    "name": "analyze_ticket",
    "description": "Analyze a support ticket",
    "input_schema": {
        "type": "object",
        "properties": {
            "issue": {"type": "string"},
            "category": {
                "type": "string",
                "enum": ["billing", "technical", "account", "other"]
            },
            "urgency": {
                "type": "string", 
                "enum": ["low", "medium", "high"]
            },
            "action_required": {"type": "boolean"},
            "next_step": {"type": ["string", "null"]}
        },
        "required": ["issue", "category", "urgency", "action_required"]
    }
}]

response = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=1024,
    tools=tools,
    tool_choice={"type": "tool", "name": "analyze_ticket"},
    messages=[{"role": "user", "content": ticket_text}]
)

# Response is now guaranteed to match the schema
result = response.content[0].input

The API enforces the schema. No JSON parsing failures.

Step 7: Test With Diverse Inputs (15 min)

Run your eval cases:

for case in eval_cases:
    result = get_summary(case["input"])
    
    # Type checks
    assert result.category in ['billing', 'technical', 'account', 'other']
    assert result.urgency in ['low', 'medium', 'high']
    
    # Content checks
    assert len(result.issue) < 500
    assert (result.action_required is False) == (result.next_step is None)

Catches both format and content issues.

Step 8: Handle Lists and Nested Structures (15 min)

For lists:

interface TicketAnalysis {
  summary: TicketSummary;
  tags: string[];
  mentioned_products: Array<{
    name: string;
    issue_type: string;
  }>;
}

LLMs handle nested structures well. Just be specific in the schema.

Step 9: Streaming Structured Output (varies)

For long structured responses, streaming is useful:

# OpenAI streaming with JSON
stream = client.chat.completions.create(
    model="gpt-4",
    stream=True,
    response_format={"type": "json_object"},
    messages=[...]
)

chunks = []
for chunk in stream:
    if chunk.choices[0].delta.content:
        chunks.append(chunk.choices[0].delta.content)

full_response = "".join(chunks)
data = json.loads(full_response)

Streaming with structure requires the API support it.

Step 10: Monitor Schema Adherence (ongoing)

In production:

• Log every parse failure

• Track failure rate per prompt version

• Investigate spikes

If failure rate exceeds 1-2%, the prompt or model is degrading.

What You Just Did

You set up reliable structured outputs. Your code consumes LLM responses as data; no parsing fragility; clear contract.

Common Failure Modes

Loose schemas. Strings where enums should be. Validation catches less.

Markdown wrapping. Model wraps JSON in json blocks. Strip or use native JSON mode.

Missing fields in production. Model occasionally omits fields. Default values or retry.

Overly complex schemas. Deeply nested structures confuse the model. Flatten.

Schema drift. Schema changes; old code breaks. Version schemas.