Tutorial 3: Few-Shot Examples That Work

Contributor
4 days ago
3 min read

Few-shot examples are the single biggest improvement most prompts get. Picking the right examples is the art. This tutorial walks through it.

What You'll Build

A small set of well-chosen few-shot examples that dramatically improve your prompt's output consistency.

Step 1: Diagnose the Need (5 min)

Few-shot helps when:

Output format is specific
The task has subtlety the description doesn't capture
Consistency across runs matters
The model often produces "almost right" outputs

Few-shot is less useful when:

The task is simple and well-described
Examples would be cherry-picked rather than representative
The task is genuinely creative (examples constrain)

Step 2: Pick 3-5 Examples (15 min)

Not 1; not 20. 3-5 is the sweet spot for most tasks.

Examples should cover:

The most common case (happy path)
A subtle case where pattern matters
An edge case that requires judgment
Optionally: a "do not do this" example

Step 3: Format Examples Consistently (10 min)

Input: [actual input]
Output: [exact desired output]

Input: [actual input]
Output: [exact desired output]

Same format every time. The model learns the format.

Tags help too:

<example>
<input>Customer: My order hasn't arrived...</input>
<output>Issue: Order #X delayed beyond ETA.</output>
</example>

Pick a convention; stick with it.

Step 4: Use Real Examples, Not Constructed (15 min)

Constructed examples ("a customer might say...") lack the messiness of real input. Use real:

Past support tickets
Real email subject lines
Actual user queries

Anonymize as needed. The realism matters.

Step 5: Show the Failure Modes (15 min)

For tasks where common mistakes matter, include a corrective example:

Input: "Customer says the app is broken but doesn't say what specifically."
Output: Issue: Customer reports app issue; specific problem not described 
in ticket. Action: Need to follow up for details.

[Note: do not invent specific issues. State explicitly when info is missing.]

The output demonstrates handling the ambiguity; the note reinforces.

Step 6: Order Matters (10 min)

The model weights recent examples more. Put the most representative example last.

For complex patterns, structure examples in increasing difficulty:

Simple case (establishes the pattern)
Slightly subtle case (shows nuance)
Edge case (shows judgment)

Step 7: Test Without Examples (10 min)

Run the prompt without examples first. See output.

Run with examples. Compare.

The delta tells you what the examples are doing. If no difference, examples aren't helping (consider removing).

Step 8: Avoid Overfitting (10 min)

Examples can overfit — the model learns the specifics rather than the pattern.

Symptoms:

Output mimics example structure too closely
Output uses words from examples not appropriate to new input
Output ignores the actual input in favor of example pattern

Fix: vary the examples more; include cases that look different from each other.

Step 9: Use Variables If Needed (10 min)

For dynamic few-shot — selecting examples per input:

def get_relevant_examples(input_query):
    # Retrieve from a database of past examples
    # Use embedding similarity or keyword match
    return matching_examples[:3]

def build_prompt(input_query):
    examples = get_relevant_examples(input_query)
    examples_text = "\n\n".join([
        f"Input: {ex['input']}\nOutput: {ex['output']}"
        for ex in examples
    ])
    return f"{system_prompt}\n\n{examples_text}\n\nInput: {input_query}\nOutput:"

Per-input examples can outperform static ones for diverse inputs.

Step 10: Maintain the Examples (ongoing)

Examples become stale:

Tone or style changes in your domain
New patterns emerge
Old patterns become irrelevant

Quarterly: review the examples. Update what's stale.

What You Just Did

You added few-shot examples that improve your prompt. Output consistency rises; edge cases get handled; the model produces what you want more reliably.

Common Failure Modes

One example. Often not enough to establish a pattern.

Too many examples. Token budget consumed; pattern gets noisy.

Cherry-picked examples. All happy path; edge cases not represented.

Constructed examples. Lack realistic messiness; production input differs.

Examples that conflict. Two examples teach different patterns; model is confused.

ShiftQuality