Structured Outputs (JSON and Beyond) — Practical Prompt Engineering, Part 4

Shawn West
Jul 8
10 min read

Updated: Aug 17

Practical Prompt Engineering · Part 4

Through Parts 1–3 our support-ticket summarizer returned prose — two clean sentences a human could skim in a queue. That's fine right up until you wire it into something. The moment you write if "high" in summary: page_oncall(), you've turned an LLM into a regex target, and it betrays you on the first ticket where "high" shows up inside the issue text — "customer is a high-value account asking about a refund" — and a billing question pages your on-call engineer at 2 a.m. Flip it around and it's worse: the model writes Urgency: High. on Monday and This one's urgent. on Tuesday, your parser catches the first and misses the second, and a genuinely hot ticket sits untouched because the string didn't match.

The problem was never the model's judgment — it's that free text has no contract. Your downstream code needs urgency == "high" as a field it can trust, not a substring it has to hunt for. This tutorial gives the summarizer that contract: a TicketSummary schema, validated with Pydantic, produced through tool/function calling so the shape is guaranteed at the API boundary instead of hoped for in a prompt. By the end you route tickets on result.urgency and never touch str.split() again.

Before you start

Have the Part 1 workspace importable as workspace.py. Step 1 imports client, MODEL, EVAL_CASES and run() from it by name, and Step 4 leans on Part 1's failure logger. If those five aren't there, fix that before you write a schema.
Have your ten EVAL_CASES loaded, including the awkward ones — the 40-message escalation thread and the no_issue thank-you note. Those are where the prompt approach cracks; a clean happy-path set will tell you everything is fine.
Python with Pydantic 2 installed. The schema uses model_validator(mode="after") and Literal, which are v2 idioms; on v1 Step 1 won't import.
The Anthropic SDK and a working key, on a model that supports tool calling. Step 3 calls client.messages.create with tools and a forced tool_choice — that's the whole guarantee, and it's an API feature, not a prompt trick.
Parts 1–3 done, so you already have a summarizer that returns good prose. This tutorial doesn't improve its judgment; it gives that judgment a contract.
Block about an hour. The labeled steps total 45 minutes, and Step 3 — the main event — is the one that isn't timed.

What you'll build

Building on the Part 1 workspace (run(), MODEL, EVAL_CASES), you'll add:

A Pydantic schema — TicketSummary — that is both your type and your validator.
Two ways to get it back: the "please output JSON" prompt (which mostly works, and I'll show exactly where "mostly" bites) and tool calling (which guarantees the shape).
A retry-repair loop that feeds the validation error back to the model instead of re-rolling the dice.
A failure-rate check wired into evaluate, so a prompt or model regression shows up as a number, not a 3 a.m. incident.

Step 1: The schema is the contract (10 min)

Write the schema as a Pydantic model, not a comment. That gives you one object that is at once the type your editor autocompletes, the validator that rejects bad data, and the documentation of what "a ticket summary" means here.

# structured.py — builds on workspace.py from Part 1
from typing import Literal, Optional
from pydantic import BaseModel, ValidationError, model_validator

from workspace import client, MODEL, EVAL_CASES, run  # Part 1 workspace


class TicketSummary(BaseModel):
    issue: str
    category: Literal["billing", "technical", "account", "other"]
    urgency: Literal["low", "medium", "high"]
    action_required: bool
    next_step: Optional[str] = None

    @model_validator(mode="after")
    def _next_step_matches_action(self):
        # A real invariant: if something must be done, say what.
        if self.action_required and not self.next_step:
            raise ValueError("action_required is true but next_step is empty")
        return self

Two decisions matter here. First, category and urgency are Literal, not str — the difference between a router you can write a match statement against and one where you're defensively lowercasing and guessing whether "med" means "medium." Second, the model_validator encodes a rule the type system alone can't: action_required and next_step have to agree. A model that says "no action needed" but fills in a next step is confused, and you want to catch that here, not three services downstream.

Step 2: The "just output JSON" prompt, and where it cracks (10 min)

The obvious approach is to ask for JSON in the prompt and parse what comes back. It works often enough to feel done, which is the trap.

import json

JSON_SYSTEM = """You are a support-ticket analyst. Respond with ONLY a JSON
object, no preamble and no markdown, matching:
{"issue": string, "category": "billing"|"technical"|"account"|"other",
 "urgency": "low"|"medium"|"high", "action_required": boolean,
 "next_step": string or null}"""


def summarize_via_prompt(ticket: str) -> TicketSummary:
    text, _ = run(JSON_SYSTEM, ticket)             # Part 1 run() -> (text, usage)
    text = text.strip()
    if text.startswith("```"):                     # strip accidental fences
        text = text.split("```")[1].removeprefix("json").strip()
    return TicketSummary(**json.loads(text))       # can raise on both lines

Run this across the ten EVAL_CASES and it'll pass eight or nine cleanly. The failures aren't random — they cluster on the messy inputs. On the 40-message escalation thread, the model gets chatty and prepends Here's the analysis: before the JSON, and json.loads throws on the first character. On the terse no_issue thank-you note, it sometimes wraps the object in a ```json fence — hence the defensive strip, which is itself a code smell: you're now maintaining a parser for the model's formatting moods. In practice this fails to parse on roughly 2–5% of realistic tickets, and every failure is a try/except you have to write, test, and babysit.

The Pydantic layer earns its keep even when the JSON is valid. If the model returns "urgency": "urgent" — a plausible word that isn't in your enum — json.loads is perfectly happy and hands you garbage; TicketSummary(**data) raises a ValidationError naming the exact field. That catch is why the schema is a validator, not a comment.

So the prompt approach can produce well-formed JSON that's still wrong, and fail to produce JSON at all. We can kill the second failure mode entirely.

Step 3: Tool calling — make the shape non-negotiable (the main event)

Here's the point of the whole tutorial, and the opinion I'll defend: for structured output, use tool/function calling, not a "please output JSON" prompt. Not because it's fancier — because it moves the guarantee from a place you can't enforce (the model's willingness to follow formatting instructions) to a place you can (the API's schema handling). Define a tool, and you hand the provider a JSON Schema and force the model to fill it in. The response comes back as a structured tool_use block whose .input is already a Python dict of the right shape — no free text to parse, no fence to strip, no preamble to skip. The whole class of "the model said something before the JSON" bugs disappears at the boundary.

Define the tool once, as JSON Schema — the same fields as your Pydantic model:

TICKET_TOOL = {
    "name": "record_ticket_summary",
    "description": "Record a structured summary of exactly one support ticket.",
    "input_schema": {
        "type": "object",
        "properties": {
            "issue": {
                "type": "string",
                "description": "One sentence naming the customer's actual problem.",
            },
            "category": {
                "type": "string",
                "enum": ["billing", "technical", "account", "other"],
            },
            "urgency": {
                "type": "string",
                "enum": ["low", "medium", "high"],
            },
            "action_required": {"type": "boolean"},
            "next_step": {
                "type": ["string", "null"],
                "description": "What support should do next, or null if nothing.",
            },
        },
        "required": ["issue", "category", "urgency", "action_required"],
    },
}

Now call the model with tool_choice forcing that tool. Forcing matters: without it the model decides whether to call the tool, and on a chatty ticket it might just answer in prose. Naming the tool in tool_choice removes the choice — every response is a tool call.

def summarize(ticket: str) -> TicketSummary:
    resp = client.messages.create(
        model=MODEL,                    # "claude-sonnet-5" from Part 1 — one line to swap
        max_tokens=512,
        tools=[TICKET_TOOL],
        tool_choice={"type": "tool", "name": "record_ticket_summary"},
        messages=[{"role": "user", "content": ticket}],
    )
    # Find the tool_use block explicitly — don't assume it's content[0].
    block = next(b for b in resp.content if b.type == "tool_use")
    return TicketSummary(**block.input)

Note the extraction: grab the tool_use block by type, not content[0]. When the model narrates, index 0 can be a text block; matching on b.type == "tool_use" never surprises you.

Point it at Part 1's delayed-shipping case and you get a real object back:

>>> summarize(EVAL_CASES[0]["input"])
TicketSummary(issue='Order #12345 has not arrived after two weeks; the '
                    'carrier delayed it and quoted a new ETA.',
              category='other',
              urgency='medium',
              action_required=True,
              next_step='Confirm the updated carrier ETA with the customer.')

That's the win in one line of output. result.urgency is "medium", guaranteed one of three values, ready for if result.urgency == "high" with no substring roulette; result.action_required is a real bool, not the string "true". Hand it straight to a router, a database row, or the next chained call in Part 5.

One honest caveat, because this is where people over-trust tool calling. The schema guarantees well-formed JSON of the right shape — object present, category a string, action_required a boolean. It does not guarantee semantic correctness: the model can still choose "account" when a human would say "billing", and depending on how strictly the provider enforces enums, occasionally emit a value just outside the set. That's why TicketSummary(**block.input) still runs through Pydantic — tool calling kills the malformed-JSON mode (down from ~2–5% to effectively zero), and Pydantic guards the invalid-value and broken-invariant modes no transport format can catch. Two layers, two jobs; drop either and you've got a gap. It's why tool calling is the default, and why the rest of this path assumes summarize() returns a trustworthy object.

Step 4: A retry loop that repairs, not just re-rolls (15 min)

Even with tool calling, Pydantic will occasionally reject an output — an enum the model drifted on, or the action_required/next_step invariant. The lazy fix is to call again and hope the sampling lands better. The senior fix is to tell the model what was wrong and let it correct itself, sending the validation error back as a tool_result marked is_error:

def summarize_safe(ticket: str, max_retries: int = 2):
    messages = [{"role": "user", "content": ticket}]
    for attempt in range(max_retries + 1):
        resp = client.messages.create(
            model=MODEL,
            max_tokens=512,
            tools=[TICKET_TOOL],
            tool_choice={"type": "tool", "name": "record_ticket_summary"},
            messages=messages,
        )
        block = next(b for b in resp.content if b.type == "tool_use")
        try:
            return TicketSummary(**block.input)
        except ValidationError as e:
            # Hand the exact error back so the next attempt is informed.
            messages.append({"role": "assistant", "content": resp.content})
            messages.append({"role": "user", "content": [{
                "type": "tool_result",
                "tool_use_id": block.id,
                "is_error": True,
                "content": f"Validation failed: {e}. "
                           f"Call record_ticket_summary again with corrected values.",
            }]})
    log_failure(ticket, block.input)   # from Part 1's logger
    return None

Watch it work on an enum-drift failure. Attempt 1 returns urgency="urgent"; Pydantic rejects it; the loop appends Validation failed: urgency Input should be 'low', 'medium' or 'high'; attempt 2 comes back with urgency="high" and validates. A blind re-roll fixes that eventually too — but on the invariant failures (action_required=true, next_step=null), naming the rule fixes it next try far more reliably than resampling, because the model didn't know the rule existed until you told it.

Cap the retries: max_retries=2 means three attempts, then None and a logged failure. A ticket that fails validation three times running usually has a real cause — a truncated input, a max_tokens cutoff mid-object — that more retries won't fix.

Step 5: Wire the failure rate into your eval (10 min)

Part 1's promise was that regressions show up as numbers. Extend evaluate to track how often validation fails — your production early-warning signal:

def evaluate_structured(cases=EVAL_CASES):
    failures = sum(summarize_safe(c["input"]) is None for c in cases)
    rate = failures / len(cases)
    print(f"validation failure rate: {rate:.1%}  ({failures}/{len(cases)})")
    return rate

Log that rate per prompt version and per model, the way Part 1 logged cost. The threshold I hold teams to: if the parse/validation failure rate climbs past ~1–2%, treat it as a regression and investigate — a prompt edit that loosened the tool description, a model version bump, or a new ticket source in a format the model hasn't seen. A rate that sat at 0.4% for a month then jumps to 3% overnight is a signal, not noise — cheaper to catch in the eval than in a router silently dropping tickets.

Which layer catches which failure

Failure mode	What it looks like	What catches it
Preamble before the JSON	Here's the analysis: and then the object; json.loads throws on the first character	Tool calling — there's no text channel to put a preamble in
Markdown fence	The object wrapped in a json fence	Tool calling; the Step 2 fence strip is a stopgap you shouldn't have to maintain
Value outside the enum	"urgency": "urgent" — valid JSON, wrong value, json.loads perfectly happy	Pydantic's Literal, then the Step 4 repair loop, which fixes it by name
Broken invariant	action_required=true with next_step=null	The model_validator, fed back as an is_error tool_result
Wrong category	Well-formed, in-enum, and still "account" where a human says "billing"	Neither layer — your eval cases and a human. Tool calling guarantees shape, not semantics

You're done when

summarize(EVAL_CASES[0]["input"]) returns a TicketSummary object, not a string: result.urgency is one of low/medium/high and result.action_required is a real bool. No json.loads, no fence stripping, no str.split() anywhere in the path.
You've watched summarize_safe repair rather than re-roll — attempt 1 comes back with something like urgency="urgent", Pydantic rejects it, the loop appends Validation failed: ..., and attempt 2 validates. Push it past three attempts and you get None plus a logged failure, not an exception.
evaluate_structured() prints a line you can paste into a log — validation failure rate: 0.0% (0/10) — and you've recorded that number against the prompt version and the model, so next week's run is a comparison instead of a guess.

Troubleshooting

Markdown-fenced JSON (prompt approach). The model wraps its output in a ```json fence and json.loads chokes on the backticks. Strip the fence as in Step 2 — or better, use the tool, where fences can't happen because there's no text channel to put them in.

StopIteration on the next(...) call. No tool_use block came back — almost always because tool_choice wasn't set to force the tool, so the model answered in prose. Set tool_choice={"type": "tool", "name": ...}. Rarely, max_tokens was too low and the call was cut off mid-object; raise it.

Missing fields. Only required fields are guaranteed present; next_step is optional by design. If a field you actually need keeps coming back absent, add it to required rather than defaulting it away in code.

Enum drift. The model emits "urgent" instead of "high", or "tech" instead of "technical". Pydantic's Literal catches it; the Step 4 repair loop fixes it by name. If it clears ~1% of cases, tighten the enum's description with an example of each value.

Schema drift across versions. You add a sentiment field; yesterday's stored summaries don't have it and old code that reads result.sentiment breaks. Give new fields a default, and version the schema (TicketSummaryV2) rather than mutating the live one.

Common mistakes

Trusting tool calling for semantics. It guarantees the shape, not that category is right. Keep Pydantic — and keep Part 1's no_issue restraint case, which catches the model inventing an issue on a thank-you note no matter how clean the JSON is.
response.content[0]. Grab the tool_use block by type, not by index; a leading text block hands you the wrong object.
Retry loops that don't repair. Re-rolling identical inputs wastes calls. Feed the validation error back as a tool_result so attempt two is smarter than attempt one.
Enums as plain strings. category: str throws away the one guarantee that makes a downstream match safe. Use Literal.
No failure-rate alarm. Untracked, a model bump can double your drop rate and you'll hear it from a customer, not a dashboard.