Few-Shot Examples That Work — Practical Prompt Engineering, Part 3

Practical Prompt Engineering · Part 3

In Part 2 you wrote a system prompt for the support-ticket summarizer, and it fixed the obvious failure — the summaries got shorter and stopped padding. But run the eval again and one case still fails: no_issue, the ticket that says "Just wanted to say thanks, great service!" The model keeps manufacturing a problem for it, because "summarize this support ticket" implies there is something to summarize. No amount of instruction wording reliably kills that reflex. I've rewritten that sentence four ways — "only report a problem if one exists," "do not invent issues" — and the model still hedges toward finding fault about a third of the time.

That is exactly the gap few-shot examples close. An instruction tells the model what you want in the abstract; an example shows it, in the model's own input/output format, what the right answer looks like on a case it's inclined to get wrong. This tutorial adds examples to the summarizer, measures the delta with the evaluate() scorer from Part 1, and spends most of its words on the two decisions that actually determine whether examples help: which examples to include, and whether to pick them fresh for every input.

Before you start

• workspace.py from Parts 1 and 2, with run(), EVAL_CASES, evaluate() and Part 2's SYSTEM_V2 all in place. Everything here is additive.

• A fresh evaluate(SYSTEM_V2) run so you know your own starting score before any examples go in. You need the before number to claim the after one.

• The same runtime as Part 1 — Python 3.10 or newer, .venv active, anthropic installed, ANTHROPIC_API_KEY in the shell, MODEL set to something your key can access.

• No new packages. The keyword-overlap selector is zero-dependency; embeddings are a later decision you should earn, not a starting one.

• A handful of real support tickets you can anonymize. Constructed "a customer might say…" examples lack the truncation, typos, and buried resolution that make real ones useful.

• Block 30–40 minutes. The static block is quick; the per-input selector is the back half.

What you'll build

A FEWSHOT block plus run_fewshot() and evaluate_fewshot() in workspace.py — demonstrations that sit in the user turn, wrapped in example, ticket and summary tags, ending on an open summary tag so the model knows the demonstrations are over. The block includes a restraint example: a thank-you note mapped to a non-summary. Then, optionally, EXAMPLE_POOL, select_examples() and build_fewshot() — a keyword-scored selector that picks examples per input instead of shipping the same three every call.

Where we left the summarizer

You already have workspace.py with run(), EVAL_CASES, and evaluate(). Part 2 produced a system prompt roughly like this:

SYSTEM_V2 = (
    "You summarize customer support tickets in one sentence. "
    "State the issue and the resolution status. "
    "Include any order or reference number. "
    "If there is no issue, say so plainly."
)

Run it and you land at 2 of 3 — delayed_shipping and one_liner pass, no_issue still fails:

[PASS] delayed_shipping (14w)  Order #12345 delayed by carrier; new ETA tomorrow, customer acknowledged.
[PASS] one_liner        (9w)   Customer requesting help resetting their password.
[FAIL] no_issue         (16w)  Customer expresses satisfaction but notes a possible concern about service consistency going forward.
2/3 passed

Read that last line closely. There is no concern about consistency anywhere in the input. The model wrote "possible concern about service consistency going forward" out of thin air, because the shape of the task pulled it toward producing a problem. This is the failure that instructions don't fix and examples do.

Add a few-shot block to the call

Examples go in the user turn, before the actual ticket, wrapped in tags so the model can tell demonstration from task. Add a helper that prepends an example block, and thread it through run():

FEWSHOT = """Here are examples of the summary style. Follow them exactly.

<example>
<ticket>Customer: Order #55912 never showed up, tracking stopped 4 days ago.
Agent: Reshipping now, overnight, no charge.</ticket>
<summary>Order #55912 lost in transit; agent reshipping overnight at no charge.</summary>
</example>

<example>
<ticket>Customer: Just wanted to say thanks, great service!</ticket>
<summary>No issue — customer sent positive feedback. Nothing to action.</summary>
</example>

"""

def run_fewshot(system: str, ticket: str, examples: str = FEWSHOT):
    user = f"{examples}<ticket>{ticket}</ticket>\n<summary>"
    return run(system, user)

The second example is the whole point of this tutorial, so let's not walk past it.

The restraint example is the one that earns its place

Most example sets are a lineup of inputs that should produce output: a shipping complaint, a refund request, a bug report. They teach format, and format was never the hard part. The case the summarizer actually fails — no_issue — is a case where the correct output is a refusal to summarize a problem, because there is no problem. You cannot teach that with more happy-path examples. You teach it by including an example whose output is restraint: an input paired with a summary that says, in your exact house format, "No issue — nothing to action."

That is the second <example> above. It is a thank-you note mapped to a non-summary. It tells the model something no instruction reliably conveys: in this system, "nothing to report" is a valid, expected answer, and here is what it looks like. One demonstration of the model saying nothing does more than three re-wordings of "don't invent issues," because the model is a pattern completer — show it the pattern of restraint once and it completes toward restraint.

To measure this instead of trusting it, run the eval both ways. Add a version of evaluate() that uses the few-shot runner:

def evaluate_fewshot(system: str, examples: str, cases=EVAL_CASES):
    passed = 0
    for c in cases:
        text, usage = run_fewshot(system, c["input"], examples)
        words = len(text.split())
        ok = (all(t.lower() in text.lower() for t in c["must_include"])
              and words <= c["max_words"])
        passed += ok
        print(f"[{'PASS' if ok else 'FAIL'}] {c['name']:16} ({words}w)  {text[:70]}")
    print(f"\n{passed}/{len(cases)} passed")
    return passed

if __name__ == "__main__":
    print("=== V2, no examples ===")
    evaluate(SYSTEM_V2)
    print("\n=== V2 + restraint example ===")
    evaluate_fewshot(SYSTEM_V2, FEWSHOT)

The delta is the entire argument for few-shot, made concrete:

=== V2, no examples ===
[PASS] delayed_shipping (14w)  Order #12345 delayed by carrier; new ETA tomorrow, customer acknowledged.
[PASS] one_liner        (9w)   Customer requesting help resetting their password.
[FAIL] no_issue         (16w)  Customer expresses satisfaction but notes a possible concern...
2/3 passed

=== V2 + restraint example ===
[PASS] delayed_shipping (13w)  Order #12345 delayed by carrier; new ETA tomorrow, customer acknowledged.
[PASS] one_liner        (9w)   Customer asking how to reset their password.
[PASS] no_issue         (11w)  No issue — customer sent positive feedback. Nothing to action.
3/3 passed

The no_issue output flipped from a fabricated concern to "No issue — nothing to action," and the two cases that were already passing did not regress. That last clause matters as much as the fix: an example that repaired no_issue but broke delayed_shipping would be a bad trade, and the only reason you know it didn't is that you ran the scorer instead of eyeballing one output. This is Part 1 paying off — the harness turns "the examples feel better" into "3 of 3, no regressions."

One warning that saved me an afternoon once: if you write the restraint example's output as a generic "N/A" or a blank line, the model over-learns it and starts answering "N/A" for real tickets that are merely short. Make the restraint output specific — name what it is (positive feedback) and what to do (nothing) — so the model learns "recognize the no-op case," not "sometimes output nothing."

How many examples: fewer than you think

Here is a claim a smart peer will argue with: for a task like this, three or four examples beat eight, and the marginal example after five is usually negative. The reasoning is mechanical, not aesthetic. Every example is tokens in the input on every call — the FEWSHOT block above is about 90 tokens, and eight rich examples run 400+, which at the Part 1 pricing you're paying on every single ticket forever. More important, examples that pile up start to conflict. Add three shipping examples and the model learns "summaries are about shipping"; feed it a billing ticket and it strains to phrase the billing issue in shipping-shaped language. That is overfitting to the examples, and it looks like the model ignoring the actual input.

My rule: one example per distinct behavior you need, not per topic. For the summarizer that's three — a normal issue with a reference number, a bare one-liner, and the restraint case — because those are the three behaviors the eval measures. Adding a fourth "angry customer" example doesn't teach a new behavior; it teaches tone the system prompt already covers. If you can't name the behavior an example uniquely teaches, cut it.

Selecting examples per input, and when it's worth it

Static examples — the same block every call — are the right default, and most summarizers should stop there. But there's a real technique for the case where they aren't enough: pick the examples per input, retrieving the ones most similar to the ticket you're about to summarize. If your tickets span billing, shipping, bugs, and account access, a fixed three-example block can only ever be representative of one or two of those, and a billing ticket gets summarized under shipping-shaped demonstrations. Dynamic selection fixes that by showing the model billing examples when the input is billing.

The honest version of this uses embedding similarity, but you can get most of the benefit from keyword overlap with zero dependencies, so start there and only reach for embeddings when you can measure that keywords aren't enough. Build a small pool of real, labeled examples and select the closest few:

EXAMPLE_POOL = [
    {"input": "Order #55912 never showed up, tracking stopped 4 days ago.",
     "output": "Order #55912 lost in transit; agent reshipping overnight at no charge.",
     "tags": {"order", "shipping", "tracking", "delay"}},
    {"input": "I was charged twice for invoice 8841 this month.",
     "output": "Duplicate charge on invoice 8841; refund of one charge required.",
     "tags": {"charge", "billing", "invoice", "refund"}},
    {"input": "Can't log in, password reset email never arrives.",
     "output": "Customer cannot log in; password reset email not delivered.",
     "tags": {"login", "password", "account", "email"}},
    {"input": "Just wanted to say thanks, great service!",
     "output": "No issue — customer sent positive feedback. Nothing to action.",
     "tags": {"praise", "thanks", "no_issue", "feedback"}},
]

def select_examples(ticket: str, pool=EXAMPLE_POOL, k: int = 3):
    words = set(ticket.lower().replace("#", " ").split())
    scored = sorted(
        pool,
        key=lambda ex: len(ex["tags"] & words),
        reverse=True,
    )
    # Always keep the restraint example in the pool's reach.
    return scored[:k]

def build_fewshot(ticket: str, k: int = 3) -> str:
    chosen = select_examples(ticket, k=k)
    blocks = "\n\n".join(
        f"<example>\n<ticket>{ex['input']}</ticket>\n"
        f"<summary>{ex['output']}</summary>\n</example>"
        for ex in chosen
    )
    return "Here are examples of the summary style. Follow them exactly.\n\n" + blocks + "\n\n"

Now build_fewshot("I was charged twice for invoice 8841") surfaces the billing example first, because its tags (charge, billing, invoice) overlap the input, and a shipping ticket surfaces the shipping example instead. Swap keyword scoring for cosine similarity over embeddings when the vocabulary drifts — customers say "money taken twice" and never the word "billing" — and the tag set stops matching. Embeddings catch the paraphrase; keywords don't. That's the one place I'd spend the dependency.

Two cautions that keep dynamic selection from backfiring. First, keep the restraint example reachable regardless of the input — a genuinely empty thank-you note shares no keywords with anything, so if your top-k drops it, no_issue regresses the moment selection turns on. Pin it, or bias the scorer to retain it. Second, dynamic selection multiplies your maintenance surface: a static block is three examples you can eyeball, a pool is dozens, and a stale or mislabeled pool example silently poisons every input that retrieves it. Don't reach for this until a static block has actually failed a case your eval measures.

Few-shot decisions, and when to revisit them

Decision	Start here	Change it when
How many examples	One per distinct behavior your eval measures — three for the summarizer	Never past five; beyond that they conflict and the model overfits to whichever topic dominates
Which behaviors	A normal issue with a reference number, a bare one-liner, and the restraint case	You can't name the behavior an example uniquely teaches — then cut it
What the restraint output says	Name what it is and what to do: "positive feedback… nothing to action"	Never write it as N/A or a blank — the model over-learns it and starts answering that for merely short tickets
Where examples come from	Real tickets, anonymized	You've run out of real ones — anonymize order numbers rather than inventing tickets
Static or per-input	One static block, identical on every call	A static block has actually failed a case your eval measures, and your tickets span distinct domains
How selection scores	Keyword overlap on a tag set, zero dependencies	Vocabulary drifts — customers say "money taken twice," never "billing" — and the tags stop matching
The restraint example under selection	Pin it, or bias the scorer to retain it	Never let top-k drop it — an empty thank-you note shares keywords with nothing, and no_issue regresses the moment selection turns on

You're done when

• Running evaluate(SYSTEM_V2) and evaluate_fewshot(SYSTEM_V2, FEWSHOT) back to back prints two scores, the second higher, and the no_issue line has flipped from a fabricated concern to No issue — customer sent positive feedback. Nothing to action. while delayed_shipping and one_liner are both still PASS.

• You feed the summarizer a short ticket that does have a real problem, and it does not come back "No issue" — proof the restraint example taught recognition of the no-op case, not "sometimes output nothing."

• tokens_in in prompt_log.jsonl rose by roughly the size of your block and no more — around 90 tokens for the FEWSHOT above. If it jumped further, you're carrying examples you can't name a behavior for.

Troubleshooting

Examples changed nothing — same pass count with and without. Either the task was already well-specified (the instruction did the job, so the examples are dead weight — remove them) or the model isn't reading them as demonstrations. Check that examples sit in the user turn before the ticket and use consistent tags; a stray format difference between example and task makes the model treat the block as content to summarize.

The model summarizes your examples instead of the ticket. Your delimiter between the last example and the real input is too weak. The <ticket>...</ticket>\n<summary> open-tag at the end of run_fewshot exists precisely so the model knows the demonstration is over and it's now completing a summary. Don't drop it.

no_issue fixed, but now short real tickets get "No issue." Overfitting to the restraint example. Its output was too generic — make it name the reason ("positive feedback"), and add one short real ticket to the pool so "short" and "no issue" stop being the same signal to the model.

Cost crept up after adding examples. Expected — examples are input tokens on every call. Check tokens_in in prompt_log.jsonl; if it jumped more than the block's size, you're probably carrying more examples than you need. Trim to one-per-behavior.

Common mistakes

• All happy-path examples. A lineup of things-to-summarize teaches format, which was never the failure. The example that earns its place is the one that teaches restraint on no_issue.

• Too many examples. Past five, they conflict and the model overfits to whichever topic dominates. One example per distinct behavior, not per topic.

• Constructed examples. A hand-written "a customer might say…" lacks the truncation, typos, and buried resolution of a real ticket. Pull from real ones and anonymize the order numbers.

• Dynamic selection too early. A pool of dozens is dozens of things to keep current. Earn it with a static block first, and only switch when the eval shows a static block can't cover your input spread.

• Never re-running the eval. Examples that helped in August can rot as your ticket mix shifts. The whole reason you built the scorer in Part 1 is so re-checking is one command, not a judgment call.

---

Continue the Practical Prompt Engineering path

• Previous — Part 2: Write Your First System Prompt

• Next — Part 4: Structured Outputs (JSON and Beyond) — turn these outputs into data your code can trust.

Related reading

• Prompt Templates for Common Tasks — reusable starting points once your examples stabilize.

• Prompt Engineering Is Just Clear Thinking — why examples beat adjectives.

• LLM Application Patterns: From Simple Completions to Reasoning Systems — where few-shot fits the bigger picture.

Part of the Practical Prompt Engineering learning path.