Prompting for Code Generation — Practical Prompt Engineering, Part 6

Practical Prompt Engineering · Part 6

The code an LLM writes in a demo and the code you can merge are two different artifacts, and the gap is exactly the part demos skip: running it. A model will hand you forty lines that look correct, import cleanly in your head, and blow up on the first pytest collection pass because it reached for a Pydantic v1 decorator on your v2 project. That's not a rare edge case — it's the median outcome for anything touching a library that shipped a major version in the last two years, because the training data blends both versions and the model picks one by coin flip.

So this tutorial is one loop, run to completion on a real task: generate → run → read the exact failure → feed it back → run again, until the tests are green and you've read them. We'll do it on the ticket summarizer you've built since Part 1 — the model writes the Pydantic validator and pytest suite for the TicketSummary output you defined in Part 4. And we'll take a position: LLMs are for boilerplate — validators, CRUD, test scaffolding, data transforms — and actively dangerous for architecture and non-obvious business logic. This task is the good kind. I'll show you why, and where the line is.

Before you start

• Have the Part 1 workspace on the path — generate.py imports run() and MODEL from it, and every prompt in this tutorial goes through that one function.

• Have the TicketSummary schema from Part 4 in front of you. Its fields and Literal values get pasted verbatim into the generation prompt with an instruction not to change them; that context is what stops the model drifting.

• Python 3.11 with Pydantic v2 and pytest actually installed, and check the installed version before you start. The v1/v2 split is the entire walkthrough — on a v1 environment the import-time failure you're meant to see never fires, and the lesson doesn't land.

• A terminal you can run pytest test_ticket_summary.py -q in, in the directory the generated files land in. Running is the gate here, not reading, and there's no substitute step.

• An API key and one session you keep open. The test-suite prompt is issued in the same session so the model can see the model file it just wrote.

• Block about an hour: two full generate → run → fix rounds, with a pytest run between each. The second round ends in a decision you make, not a fix the model applies.

What you'll build

Two files, both generated, both verified by running them:

1. ticket_summary.py — the TicketSummary model from Part 4, hardened with two rules: issue can't be blank, and next_step is required whenever action_required is True.

2. test_ticket_summary.py — a pytest suite covering the happy path, both rules, and the rejection cases.

The point isn't the code. It's the loop that turns "looks right" into "is right."

Why this task is the right shape for an LLM

A validator is fully specified by its schema and two written rules. No design decision, no trade-off, no place where a wrong guess costs you a week — the worst case is a test that fails loudly. That's the tell for LLM-appropriate work: the spec is complete before you start, and failure is cheap and visible. CRUD endpoints, DTO mappers, fixture factories, a datetime-to-ISO transform — same shape. You know exactly what correct looks like; you just don't want to type it.

Contrast "decide how we should score ticket priority." Opposite shape — the spec is the hard part, and a plausible wrong answer ships silently. Hold that thought; it's the line I defend at the end.

Write the generation prompt

Reuse the run() function from your Part 1 workspace. The whole job is loading the prompt with enough context that the model can't drift: the language and exact library version, the existing code it's extending, and the rules as testable statements, not vibes.

# generate.py — uses run() and MODEL from your Part 1 workspace
from workspace import run, MODEL   # MODEL = "claude-sonnet-5"

GEN_PROMPT = """You are writing Python 3.11 for a project on Pydantic v2.11.
Style: type hints on everything, no comments except for non-obvious logic.

Here is the existing model from our structured-output layer (do not change
its fields or the Literal values):

class TicketSummary(BaseModel):
    issue: str
    category: Literal['billing', 'technical', 'account', 'other']
    urgency: Literal['low', 'medium', 'high']
    action_required: bool
    next_step: Optional[str]

Add exactly two validation rules:
1. `issue` must not be blank or whitespace-only. Strip it and reject if empty.
2. `next_step` is required (non-empty) when `action_required` is True;
   it may be None otherwise.

Return the full ticket_summary.py file. Pydantic v2 only."""

code, usage = run("You are a careful Python engineer.", GEN_PROMPT)
open("ticket_summary.py", "w").write(code)
print("wrote ticket_summary.py", usage)

Naming Pydantic v2.11 twice isn't redundant — it's the most load-bearing token in the prompt, and the model ignores it anyway.

Generate, then run it — this is the whole tutorial

Here's what a first generation realistically comes back with. Read it before you run it, but don't trust your reading — this is where people lose the afternoon.

# ticket_summary.py  (generated — first pass)
from typing import Literal, Optional
from pydantic import BaseModel, validator, root_validator


class TicketSummary(BaseModel):
    issue: str
    category: Literal["billing", "technical", "account", "other"]
    urgency: Literal["low", "medium", "high"]
    action_required: bool
    next_step: Optional[str] = None

    @validator("issue")
    def issue_not_blank(cls, v):
        if not v.strip():
            raise ValueError("issue must not be blank")
        return v.strip()

    @root_validator
    def next_step_required(cls, values):
        if values.get("action_required") and not values.get("next_step"):
            raise ValueError("next_step required when action_required is True")
        return values

If you eyeball that, it looks fine. The rules are both there. The logic is correct. And it is broken in a way no amount of reading reliably catches — @validator and @root_validator are Pydantic v1 decorators. On your v2 project, a bare @root_validator doesn't warn and limp along; it refuses to build the class at import time. You only learn that by running it.

Ask for the tests too, in the same session so the model sees the model file:

TEST_PROMPT = """Write a pytest suite for the TicketSummary above in
test_ticket_summary.py. Use a make(**overrides) factory. Cover: a valid
summary; blank issue rejected; action_required=True with next_step=None
rejected; action_required=False with next_step=None allowed; an unknown
category rejected."""

A realistic generated test_ticket_summary.py:

import pytest
from pydantic import ValidationError
from ticket_summary import TicketSummary


def make(**overrides):
    base = dict(
        issue="Card declined at checkout",
        category="billing",
        urgency="high",
        action_required=True,
        next_step="Retry the payment and email the customer",
    )
    base.update(overrides)
    return base


def test_valid_summary_parses():
    assert TicketSummary(**make()).category == "billing"

def test_blank_issue_rejected():
    with pytest.raises(ValidationError):
        TicketSummary(**make(issue="   "))

def test_action_required_needs_next_step():
    with pytest.raises(ValidationError):
        TicketSummary(**make(action_required=True, next_step=None))

def test_no_action_allows_null_next_step():
    s = TicketSummary(**make(action_required=False, next_step=None))
    assert s.next_step is None

def test_unknown_category_rejected():
    with pytest.raises(ValidationError):
        TicketSummary(**make(category="general"))

def test_category_is_case_insensitive():
    assert TicketSummary(**make(category="Billing")).category == "billing"

Now run it. Not read it — run it.

pytest test_ticket_summary.py -q

==================================== ERRORS ====================================
___________________ ERROR collecting test_ticket_summary.py ____________________
ticket_summary.py:7: in <module>
    class TicketSummary(BaseModel):
E   pydantic.errors.PydanticUserError: If you use `@root_validator` with
E   pre=False (the default) you MUST specify `skip_on_failure=True`. Note that
E   `@root_validator` is deprecated and should be replaced with `@model_validator`.
=========================== 1 error in 0.11s ===========================

Zero tests ran. The module can't even be imported, so pytest never gets to a single assertion. This is the failure that reading past — it's the whole reason "does it look right" is worthless as a gate. The value of the run is that the error message is specific: it names the decorator, the version problem, and the replacement.

So use it. This is the targeted-iteration pattern, and it beats regenerating from scratch every time: paste the exact error back and ask for the specific fix, nothing more.

FIX_PROMPT = """Running the file raises at import time:

pydantic.errors.PydanticUserError: If you use `@root_validator` with pre=False
(the default) you MUST specify `skip_on_failure=True`. Note that `@root_validator`
is deprecated and should be replaced with `@model_validator`.

Rewrite ticket_summary.py using Pydantic v2 idioms only: @field_validator for
the issue check and @model_validator(mode="after") for the cross-field rule.
Change nothing else."""

Regenerating from a blank prompt would just reshuffle the v1/v2 dice; pinning the model to the one error keeps every correct line intact. The corrected file:

# ticket_summary.py  (corrected — v2 idioms)
from typing import Literal, Optional
from pydantic import BaseModel, field_validator, model_validator


class TicketSummary(BaseModel):
    issue: str
    category: Literal["billing", "technical", "account", "other"]
    urgency: Literal["low", "medium", "high"]
    action_required: bool
    next_step: Optional[str] = None

    @field_validator("issue")
    @classmethod
    def issue_not_blank(cls, v: str) -> str:
        if not v.strip():
            raise ValueError("issue must not be blank")
        return v.strip()

    @model_validator(mode="after")
    def next_step_required(self) -> "TicketSummary":
        if self.action_required and not (self.next_step and self.next_step.strip()):
            raise ValueError("next_step is required when action_required is True")
        return self

Run again:

collected 6 items

test_ticket_summary.py .....F                                            [100%]

=================================== FAILURES ===================================
______________________ test_category_is_case_insensitive _______________________
E   pydantic_core._pydantic_core.ValidationError: 1 validation error for TicketSummary
E   category
E     Input should be 'billing', 'technical', 'account' or 'other'
E     [type=literal_error, input_value='Billing', input_type=str]
========================= 1 failed, 5 passed in 0.08s ==========================

Five green, one red — and here's the part you'd have missed if you only skimmed the tests: the failing test is the model's fault, not the code's. Nothing in your two rules said categories are case-insensitive. The model invented test_category_is_case_insensitive on its own, asserting that "Billing" should coerce to "billing". The code correctly rejects it, because a Literal is exact-match. So the failure is a disagreement between two things the model made up — a hallucinated spec in the test and the honest behavior of the code.

That's the moment the run earns its keep a second time. A green suite would have hidden this; instead you're forced to make the call the model isn't allowed to make: do we actually want case-insensitive categories? In this system, category comes from our own classifier and is always canonical — the normalization is scope creep for an input that can't occur. So the test is deleted, not satisfied. If you'd wanted case-insensitivity, you'd add a field_validator on purpose and keep the test. Either way a human decided; the model just surfaced the question. Re-run, six-minus-one, all green:

collected 5 items
test_ticket_summary.py .....                                            [100%]
========================= 5 passed in 0.07s ==========================

Two iterations, both driven by an error the model produced and then fixed, neither one a from-scratch regeneration. That loop — not the prompt wording — is the skill.

Where LLM code actually goes wrong

The v1 decorator was one instance of a general pattern. Recurring failure modes, in the order they hit:

• Outdated patterns. The one you just watched: v1 Pydantic on a v2 project, datetime.utcnow() (deprecated in 3.12) instead of datetime.now(timezone.utc), old React lifecycle methods. Anything with a major version bump in the training window is a coin flip.

• Made-up library APIs. The model confidently calls TicketSummary.parse_raw_json() — a method that does not exist. AttributeError at runtime, which is why runtime beats reading.

• Insecure patterns, stated plainly. Ask for a "look up the ticket by id" DB helper and you'll get cursor.execute(f"SELECT * FROM tickets WHERE id = {ticket_id}") about a third of the time — textbook SQL injection, because the corpus is full of exactly that string. It runs. It passes a happy-path test. It's a breach waiting for one crafted ticket_id, and it never announces itself with an error — you catch it only in review.

• Over-defensive noise. try/except around a dict literal that cannot throw, null checks the type system already rules out. Not dangerous, just clutter that hides the real logic.

The v2 fix was cheap because the tooling caught it. The SQL injection is dangerous precisely because nothing catches it for you — the argument for the next section.

Where the line is, and why I hold it there

The validator worked because the spec was complete before the model started and every mistake was loud — an import error, a failing assertion. Now hand it the priority-scoring rule instead: "weight urgency against customer tier and SLA clock." The model produces a clean, well-typed function that passes the tests it also writes — grading its own homework against its own guess at the spec. And it quietly weights a mid-tier customer past an enterprise one whose SLA expires in an hour: a business error that costs a renewal, ships green, and surfaces three weeks later in a churn report.

That's the line. Not "hard code" versus "easy code" — spec-complete, loud-failure work versus judgment work where a wrong answer is invisible. Validators, CRUD, fixtures, transforms sit on the safe side: you see the failure the same day. Architecture, pricing, auth boundaries, non-obvious domain rules sit on the other, and no model-written suite saves you there, because the specification itself is what's uncertain. Generate the first kind all day. Write the second yourself, and let the model only type up a decision you've already made.

Failure modes, and how loudly each one announces itself

Failure mode	What it looked like here	How it surfaces
Outdated patterns	@validator and @root_validator on a v2 project; datetime.utcnow() in place of datetime.now(timezone.utc)	Loudest possible — PydanticUserError at import, so zero tests run
Made-up library API	TicketSummary.parse_raw_json(), a method that does not exist	AttributeError at runtime; reading the line never flags it
Insecure patterns	cursor.execute(f"SELECT * FROM tickets WHERE id = {ticket_id}")	Silent. It runs, it passes the happy-path test, and only review catches it
Over-defensive noise	try/except around a dict literal that cannot throw; null checks the types already rule out	Silent and harmless — clutter that hides the real logic
Invented spec in the tests	test_category_is_case_insensitive, a rule nobody asked for	A red test that is the test's fault, not the code's — visible only if you read the assertions

You're done when

• pytest test_ticket_summary.py -q gets past collection. The first run errors before a single assertion executes; you're done when the module imports and the run reports 5 passed.

• ticket_summary.py contains @field_validator and @model_validator(mode="after") and no @validator or @root_validator anywhere — and you got there by pasting the exact error into a fix prompt, with every already-correct line still intact, not by regenerating the file.

• Your suite is five tests, not six, and test_category_is_case_insensitive is gone because you decided categories are exact-match — not because you added normalization to make the code satisfy a rule you never specified.

Troubleshooting

PydanticUserError at import / pytest collection error. You're on the v1-decorator problem from the walkthrough. Paste the whole error into the fix prompt and demand @field_validator / @model_validator(mode="after"). Don't regenerate the file from scratch.

All tests pass on the first run. Be suspicious, not relieved. A six-line model plus a model-written suite going green immediately often means the tests only cover what the code happens to do. Read the assertions against your spec — that's how test_category_is_case_insensitive gets caught.

AttributeError: ... has no attribute at runtime. A hallucinated API. The method doesn't exist. Check the library's actual reference and paste the real signature into a fix prompt.

The model keeps drifting back to v1 across a long session. Its context is diluting. Restate Pydantic v2 only, no @validator or @root_validator in the fix prompt rather than trusting the original instruction to hold.

Common mistakes

• Reading instead of running. The v1 decorator, the made-up method, and the SQL injection all read fine. Only two of the three even error, and only when executed.

• Regenerating instead of iterating. Starting over reshuffles the v1/v2 dice and throws away correct lines. Paste the exact error; ask for the one fix.

• Trusting model-written tests as a spec. They encode the model's guess at what you meant, sometimes a rule you never asked for. A passing suite from the same model that wrote the code is grading its own homework.

• Handing over judgment work. A validator failing loudly is cheap; a priority rule failing silently is a lost renewal. Keep invisible-failure work on your side of the line.

---

Continue the Practical Prompt Engineering path

• Previous — Part 5: Chain Prompts to Break Down Hard Tasks

• Next — Part 7: Multi-Turn Conversations — hold context across many turns without losing the thread.

Related reading

• What LLMs Get Wrong About Code (and How to Catch It) — the failure catalog to review against.

• Prompt Engineering Is Just Clear Thinking — why specific requests produce specific code.

• LLM Application Patterns: From Simple Completions to Reasoning Systems — fitting generated code into a real system.

Part of the Practical Prompt Engineering learning path.