Write Your First System Prompt — Practical Prompt Engineering, Part 2

Shawn West
Jul 8
10 min read

Updated: Aug 17

Practical Prompt Engineering · Part 2

Part 1 ended on a number that should bother you: 2 out of 3. The workspace ran a one-line system prompt — You summarize customer support tickets in one or two sentences. — against three fixed cases, and two of them failed in ways you'd never have caught by eyeballing a single ticket. The one_liner case ("How do I reset my password?") came back padded to 22 words of corporate throat-clearing, blowing the 30-word budget. The no_issue case — a customer writing in just to say thanks — came back with a fabricated complaint about "service quality" that appears nowhere in the ticket.

Those two failures are the entire assignment for this tutorial. We're going to write a real system prompt — call it SYSTEM_V2 — that fixes exactly those two behaviors, wire it into the workspace you already built, and re-run evaluate() to watch the score go from 2/3 to 3/3. No new framework, no rewrite. The harness from Part 1 does the grading; our only job is the prompt.

Before you start

workspace.py from Part 1, working — run(), EVAL_CASES and evaluate() all defined. This tutorial adds a string to the bottom of that file; it doesn't replace anything.
Part 1's baseline in front of you: SYSTEM_V1 scoring 2/3, with one_liner padded past 30 words and no_issue inventing a complaint. Those two failures are the assignment.
The same runtime as Part 1 — Python 3.10 or newer, your .venv activated, anthropic installed, and ANTHROPIC_API_KEY set in the shell you're actually running in.
A MODEL string your key can access. The samples use claude-sonnet-5; swap it if yours is different.
Be ready to add temperature=0 to the create() call in run(). An eval that swings between runs hasn't been fixed, it's been made lucky.
Block about 30 minutes. There's one prompt to write and one command to re-run.

What you'll build

SYSTEM_V2 — a single Python string in workspace.py that replaces Part 1's one-line placeholder with a real system prompt: role, task with the word budget stated, input shape, output contract, constraints, and an explicit refusal case. Every clause in it traces to a case your harness already grades. When it's in, evaluate() takes the summarizer from 2/3 to 3/3 without touching the harness.

The lever you're actually pulling

A system prompt isn't an incantation and it isn't personality. It's the instruction block the model reads before it ever sees the user's message, and it stays in force for every turn of the conversation. That position is the whole point: it's the one place you define the job once and have it apply to every ticket — angry, empty, or a 40-message thread with the resolution buried in the middle.

Here's the opinion that will save you the most time, and that plenty of prompt-guide authors would fight me on: the "You are a world-class senior support analyst with 20 years of experience" opener is mostly wasted tokens. Role-play framing changes vocabulary — it'll nudge a model toward legalese or toward casual — but it does almost nothing to change behavior under pressure. What stops the model from inventing a complaint on a thank-you note isn't a more impressive job title. It's a constraint that says don't, plus an explicit instruction for what to do instead. Constraints steer; flattery decorates.

Anatomy of a prompt that steers behavior

Six things go into a system prompt that holds up on real inputs. They are not equally important, and treating them as a tidy six-item checklist is exactly how people end up with a 400-word prompt that still fails on the thank-you note. Here they are in rough priority order, with the two that fix our failing cases getting most of the ink.

1. Role — one line, then stop. You are a support-ticket summarizer for an internal analytics pipeline; your output is read by dashboards and other code, never by the customer. The last clause is doing real work: it tells the model there's no human on the other end to be polite to, which kills the "Great question!" reflex and the urge to soften bad news. One sentence, and it earns its place by changing who the audience is — not by flattering the model.

2. Task — the specific verb and the shape. Summarize the ticket in one sentence of 30 words or fewer, capturing the specific issue and its current resolution status. Notice the 30-word limit is in the prompt, not only in the eval. Part 1's evaluate() rejects anything over max_words: 30, but the model never saw that rule — you were grading it against a constraint you'd kept secret. This one line is the direct fix for one_liner. Tell the model the budget it's being scored against.

3. Inputs — what it'll receive. Cheap to state, and it stops the model from treating a multi-turn transcript as a single customer utterance: You'll receive a transcript with "Customer:" and "Agent:" turns. One clause, done.

4. Output contract — where downstream code lives or dies. This is the most underrated line in the whole prompt. Concrete failure: suppose 40% of your summaries come back wrapped as Here's the summary: "Order #12345…". Your dashboard code does len(text.split()) to check the word count — and now it's counting Here's, the, summary:, and two quotation marks as content. Cases that produce a perfectly good 26-word summary start failing at 31 words for a reason that has nothing to do with the summary. So: Output only the summary. No preamble, no labels, no quotation marks. If anything reads your output programmatically, this line is not optional.

5. Constraints and the refusal case — spend the most here. This is what fixes no_issue, and it's the part most first prompts get wrong. The reason a weak prompt hallucinates a complaint on a thank-you note is subtle: the instruction "summarize this ticket" presupposes there is something to summarize. Faced with a ticket that says "thanks, great service!", the model tries to be useful and manufactures the missing issue to fill the shape you asked for. You don't fix that by adding "don't hallucinate" — that's a wish, not an instruction. You fix it by giving the model an explicit, sanctioned way to report nothing:

If the transcript contains no problem, complaint, or request — for
example a thank-you note — do not invent one. Respond with exactly:
No customer issue to summarize.

Two things make that line work. First, it names the failure mode ("do not invent one"), so the model recognizes the situation instead of pattern-matching to "produce a summary." Second, it hands over an exact string. That's easy for the model to comply with, and — not by accident — it's exactly what Part 1's no_issue check looks for: must_include: ["no", "issue"]. No customer issue to summarize. contains both. A vague escape hatch ("mention if there's no issue") produces vague, unverifiable output that your substring check can't confirm. Give the refusal a shape you can grade.

One more constraint, the general form of the same discipline: Use only facts stated in the transcript; do not infer causes or outcomes. That stops the model from padding the delayed-shipping summary with a guessed root cause it was never told.

6. Examples — optional, and deliberately last. One to three worked examples (few-shot) are the highest-leverage addition for format-heavy tasks, and they're the entire subject of Part 3. But reaching for examples first is a common over-correction. For a task this constrained, a precise instruction plus a refusal clause gets you to 3/3 with zero examples — and every example you add is more tokens on every single call. Add examples when instructions genuinely can't pin down the format, not as a reflex.

Assemble it

Put the pieces together as one Python string. Add it to the bottom of the workspace.py you built in Part 1 — it already imports the SDK, sets MODEL, and defines run, evaluate, and EVAL_CASES.

# workspace.py  (MODEL = "claude-sonnet-5" — swap for whatever your key can access)

SYSTEM_V2 = """\
You are a support-ticket summarizer for an internal analytics pipeline.
Your output is read by dashboards and other code, never by the customer.

You'll receive a transcript with "Customer:" and "Agent:" turns.
Summarize it in one sentence of 30 words or fewer, capturing the specific
issue and its current resolution status. Preserve identifiers such as order
numbers exactly as written.

If the transcript contains no problem, complaint, or request — for example a
thank-you note — do not invent one. Respond with exactly:
No customer issue to summarize.

Rules:
- Output only the summary. No preamble, no labels, no quotation marks.
- Use only facts stated in the transcript; do not infer causes or outcomes.
- Third person. No first-person narration ("I see that...").
"""

Every line there traces to a specific case: the 30-word limit is for one_liner, the refusal clause is for no_issue, "preserve identifiers exactly" keeps the 12345 that delayed_shipping checks for, and "output only the summary" keeps the word count honest for all three.

Prove it: 2/3 → 3/3

Don't trust the prose — trust the harness. Point evaluate() at the new prompt:

if __name__ == "__main__":
    print("=== V2 ===")
    evaluate(SYSTEM_V2)

python workspace.py

=== V2 ===
[PASS] delayed_shipping (14w)  Order #12345 delayed by a carrier issue; new ETA tomorrow, customer acknowledged.
[PASS] one_liner       (11w)  Customer asks how to reset their account password; no resolution provided yet.
[PASS] no_issue        ( 5w)  No customer issue to summarize.
3/3 passed

There's the whole tutorial in three lines. one_liner dropped from 22 words to 11 and kept password. no_issue stopped inventing a complaint and returned the exact refusal string, which satisfies ["no", "issue"]. delayed_shipping, which already passed, still passes — the change didn't regress it. That last part is why Part 1 insisted on fixed cases: a "fix" that quietly breaks a case that used to work is the most expensive kind of change, and the only way to see it is to re-run the constant set.

Want to watch the specific behavior change instead of just the score? Use Part 1's compare() on the case that mattered:

compare(SYSTEM_V1, SYSTEM_V2, cases=[EVAL_CASES[2]])   # no_issue

=== no_issue ===
  A: The customer reports an issue with the service quality they received.
  B: No customer issue to summarize.

A invents a problem to be helpful; B correctly reports that there's nothing to summarize. That side-by-side, on a thank-you note, is the difference the refusal clause bought you — and it's the kind of failure that never shows up if your only test input is a ticket that genuinely has a problem.

Because you're calling evaluate(), every one of these runs also appended to prompt_log.jsonl. Three weeks from now, grep no_issue prompt_log.jsonl shows the exact moment this case flipped from fabrication to refusal. That's your version history — no separate ticket_summary_v3.txt folder required. The named SYSTEM_V2 constant plus the log is the audit trail.

What each clause is fixing

Prompt component	The line doing the work	What it fixes
Role — one line, then stop	"your output is read by dashboards and other code, never by the customer"	Kills the politeness reflex; no résumé needed
Task — verb plus shape	"one sentence of 30 words or fewer"	one_liner — stops you grading against a rule the model never saw
Inputs — what it receives	a transcript with "Customer:" and "Agent:" turns	A multi-turn thread read as a single customer utterance
Output contract	"Output only the summary. No preamble, no labels, no quotation marks."	Keeps len(text.split()) honest; a good 26-word summary failing at 31
Constraint — identifiers	"Preserve identifiers such as order numbers exactly as written"	delayed_shipping — the model normalizing #12345 into prose
Constraint — the refusal case	"do not invent one. Respond with exactly: No customer issue to summarize."	no_issue — names the failure mode and hands over a gradeable string
Constraint — no inference	"Use only facts stated in the transcript; do not infer causes or outcomes"	Padding a summary with a guessed root cause
Examples	None here	Part 3's tool — a constrained task reaches 3/3 without paying tokens for them

You're done when

evaluate(SYSTEM_V2) ends with 3/3 passed, and delayed_shipping is still PASS — the fix didn't cost you the case that already worked.
The no_issue line reads exactly No customer issue to summarize. — the string from your prompt, not a paraphrase of it.
You run it twice and get the same score both times, and no summary opens with Here's the summary: — so the word counts in the (Nw) column are counting summary, not preamble.

Troubleshooting

The model still opens with "Here's the summary:". The output-contract line is present but buried. Make it its own line near the end of the prompt where trailing instructions get the most weight, and if preamble persists, prefill the assistant turn (start its response for it) so there's no room for a greeting. Preamble is the single most common reason a good summary fails a word-count check.

no_issue still fabricates a complaint. Either your refusal string doesn't match what must_include checks, or the model paraphrased it ("There is no customer issue to summarize here."). Pin the exact string in the prompt, and set temperature=0 in run()'s create() call — paraphrasing on a fixed instruction is usually temperature noise.

one_liner still runs long. LLMs treat "30 words" as a soft target, not a hard cap; expect it to land at 25–35, not exactly 30. If you need a genuine ceiling, enforce it in code (truncate, or set a low max_tokens as a backstop) rather than trusting the model to count. The prompt gets you close; code makes it exact.

delayed_shipping loses the order number. The model normalized #12345 to "the customer's order." That's why "preserve identifiers exactly as written" is its own clause — without it, models routinely round-trip IDs into prose.

Scores swing between runs. You're running at a non-zero temperature. For an eval you want determinism, so add temperature=0 to the client.messages.create(...) call in run(). A prompt that passes 3/3 on one run and 2/3 on the next hasn't been fixed; it's been made lucky.

Common mistakes

Grading against a rule the model never saw. The 30-word limit lived only in evaluate() in Part 1. If it's not in the prompt too, you're penalizing the model for missing a target you hid from it.
"Don't hallucinate" as a constraint. It's a wish. The working version names the situation and gives an exact fallback string — that's what turned no_issue around.
A role line that keeps going. "World-class expert with 20 years of experience" doesn't steer behavior; the one clause that does ("read by code, never the customer") is worth more than the whole résumé.
Reaching for few-shot examples first. Tempting, and it's Part 3's tool — but for a constrained task, examples are tokens you pay on every call to solve a problem one constraint already solves.
No output contract. Skip "output only the summary" and downstream split(), json.loads(), or regex breaks on preamble and quotation marks the model helpfully added.

Continue the Practical Prompt Engineering path

Previous — Part 1: Set Up Your Prompting Workspace
Next — Part 3: Few-Shot Examples That Work — where examples fix the failures a plain instruction can't.