Set Up Your Prompting Workspace — Practical Prompt Engineering, Part 1

S. West

Jul 8

9 min read

Updated: 4 days ago

Practical Prompt Engineering · Part 1

Most people "do prompt engineering" by typing into a chat window, squinting at the answer, tweaking a word, and running it again. It feels like progress. It isn't — because you're testing one prompt against one input and judging it by vibes. Change the input and the "better" prompt you just landed on quietly falls apart. You have no way to know, so a week later you rediscover the same failure and fix it the same way. That loop is the single biggest reason prompting feels like superstition instead of engineering.

This tutorial fixes the loop, not the prompt. By the end you'll have a small workspace — about 120 lines of Python — that runs any prompt against a fixed set of test cases, shows you the outputs side by side, tracks what each call cost, and writes every run to a log. Once that exists, every later tutorial in this path gets faster, because you can change one thing and measure whether it helped.

What you'll build

A single-file harness, workspace.py, that gives you four things the chat window can't:

Fixed test cases — the same 5–10 inputs every time, so a change is measured against a constant.
A side-by-side runner — two prompt versions, same inputs, outputs next to each other.
Cost and token tracking — so the bill is never a surprise.
A JSONL log — an append-only record of everything you tried, because your memory of "the version that worked on Tuesday" is not reliable.

We'll build it around one running example — a customer-support-ticket summarizer — and reuse that example through the rest of the path.

Prerequisites (10 min)

You need three things:

Python 3.10 or newer. Check with python --version.
An API key from a model provider. This tutorial uses Anthropic's API; the structure is identical for OpenAI or a local model behind an OpenAI-compatible endpoint — only the client import and model name change.
The SDK, installed into a fresh virtual environment so you don't pollute your system Python:

python -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install anthropic

Now put your key in an environment variable. Do not paste it into the code — hardcoded keys are the number-one way secrets end up committed to a public repo. The original habit of writing api_key="sk-..." in a script has leaked more credentials than any breach.

export ANTHROPIC_API_KEY="your-key-here"    # Windows PowerShell: $env:ANTHROPIC_API_KEY="your-key-here"

The SDK reads that variable automatically. Nothing secret ever touches the file.

Step 1: One reliable call (10 min)

Start with the smallest thing that can work — a function that takes a system prompt and a user message, calls the model, and hands back both the text and the usage numbers. Everything else in the workspace is built on this.

# workspace.py
import os
import anthropic

MODEL = "claude-sonnet-5"   # swap for whatever model your key can access
client = anthropic.Anthropic()   # reads ANTHROPIC_API_KEY from the environment

def run(system: str, user: str, max_tokens: int = 512):
    """One call. Returns (text, usage)."""
    resp = client.messages.create(
        model=MODEL,
        max_tokens=max_tokens,
        system=system,
        messages=[{"role": "user", "content": user}],
    )
    return resp.content[0].text, resp.usage

if __name__ == "__main__":
    text, usage = run("You are a terse assistant.", "What is 2 + 2?")
    print(text)
    print(usage)

Run it:

python workspace.py

You should see something like:

4
Usage(input_tokens=19, output_tokens=5)

If you got that, the plumbing works and you have a baseline. If you got an error instead, jump to Troubleshooting — three of the four errors you'll ever hit show up on this first call.

Step 2: Fixed test cases — the part that actually matters (20 min)

This is the heart of the workspace, and it's the step people skip. Without fixed cases, you are not iterating; you are guessing.

Here's why it matters with a concrete number. Say you're summarizing support tickets and you tweak your prompt. You try it on one ticket, it reads better, you ship it. But your tickets aren't one shape — some are angry, some are one line, some are a 40-message thread where the resolution is buried in the middle. A prompt tuned on the easy ticket can regress on the hard ones and you'd never see it. Ten fixed cases turn "it looks better" into "it improved 7 of 10 and broke 1" — which is a decision you can actually make.

Write your cases as data, not prose. Each one is an input plus something checkable about the expected output:

EVAL_CASES = [
    {
        "name": "delayed_shipping",
        "input": "Customer: Order #12345 hasn't arrived, it's been 2 weeks.\n"
                 "Agent: Carrier delay, new ETA tomorrow.\nCustomer: Ok thanks.",
        "must_include": ["12345", "delay"],
        "max_words": 30,
    },
    {
        "name": "one_liner",
        "input": "Customer: How do I reset my password?",
        "must_include": ["password"],
        "max_words": 30,
    },
    {
        "name": "no_issue",   # a ticket with no real problem — should NOT invent one
        "input": "Customer: Just wanted to say thanks, great service!",
        "must_include": ["no", "issue"],
        "max_words": 30,
    },
]

Notice the last case. Most eval sets only include inputs that should produce output. The one that catches real bugs is the input that should produce restraint — the ticket with no issue, where a weak prompt will hallucinate a problem to be helpful. Include the cases that test what the model shouldn't do, not just what it should.

Step 3: A runner that scores, not just prints (20 min)

Now run every case through a prompt and check it against the expectations. The check is deliberately crude — substring and word-count — because a crude automatic check you'll actually run beats a perfect one you won't.

def evaluate(system: str, cases=EVAL_CASES):
    passed = 0
    for c in cases:
        text, usage = run(system, c["input"])
        words = len(text.split())
        ok_include = all(t.lower() in text.lower() for t in c["must_include"])
        ok_length = words <= c["max_words"]
        ok = ok_include and ok_length
        passed += ok
        flag = "PASS" if ok else "FAIL"
        print(f"[{flag}] {c['name']:16} ({words}w)  {text[:70]}")
    print(f"\n{passed}/{len(cases)} passed")
    return passed

Point it at a first-draft system prompt and see where you stand:

SYSTEM_V1 = "You summarize customer support tickets in one or two sentences."

if __name__ == "__main__":
    evaluate(SYSTEM_V1)

A realistic first run:

[PASS] delayed_shipping (18w)  Order #12345 was delayed by the carrier; new delivery ETA is tomorrow.
[FAIL] one_liner       (22w)  The customer is requesting assistance with resetting their account password...
[FAIL] no_issue        (12w)  The customer reports an issue with the service quality they received.
2/3 passed

Look at what the failures tell you. The one-liner got padded past your limit; the thank-you note got a fabricated issue. Neither of those is visible from testing a single happy-path ticket. This is the whole point — the workspace made two real weaknesses show up in five seconds, and you now know exactly what Part 2's system prompt has to fix.

Step 4: Compare two versions side by side (15 min)

When you change a prompt, you want to see the old and new outputs on the same input, together. Add a comparison runner:

def compare(system_a: str, system_b: str, cases=EVAL_CASES):
    for c in cases:
        a, _ = run(system_a, c["input"])
        b, _ = run(system_b, c["input"])
        print(f"=== {c['name']} ===")
        print(f"  A: {a}")
        print(f"  B: {b}\n")

This is the difference between "I think the new prompt is better" and "on the no_issue case, A invents a problem and B correctly says there's nothing to summarize." You'll lean on compare constantly once you're tuning wording in Part 2 and beyond.

Step 5: Track what it costs (15 min)

Every call has a price, and it's easy to burn real money running a 200-case eval in a loop without noticing. Wrap the usage numbers in a cost estimate. Pricing changes often — put the per-token rates in one place and treat them as the one thing you must keep current:

# Dollars per 1M tokens — CHECK YOUR PROVIDER'S CURRENT PRICING and update these.
PRICE_IN_PER_M = 3.00
PRICE_OUT_PER_M = 15.00

def cost(usage) -> float:
    return (usage.input_tokens * PRICE_IN_PER_M
            + usage.output_tokens * PRICE_OUT_PER_M) / 1_000_000

Print a running total at the end of an eval and you'll never be surprised by a bill again. On a 10-case summarizer eval this is fractions of a cent — but the same harness pointed at a 500-case regression suite with long inputs is real money, and you want the number before the invoice.

Step 6: Log every run (15 min)

Your memory of what you tried last week is worse than you think. A one-line-per-run log is the cure — append-only JSONL you can grep, diff, or load into a notebook later:

import json, pathlib
from datetime import datetime, timezone

LOG = pathlib.Path("prompt_log.jsonl")

def log_run(system: str, user: str, output: str, usage):
    LOG.open("a", encoding="utf-8").write(json.dumps({
        "ts": datetime.now(timezone.utc).isoformat(),
        "model": MODEL,
        "system": system[:200],
        "input": user[:200],
        "output": output,
        "tokens_in": usage.input_tokens,
        "tokens_out": usage.output_tokens,
        "cost": round(cost(usage), 6),
    }) + "\n")

Call log_run from inside evaluate and every experiment leaves a trace. Three weeks from now, grep no_issue prompt_log.jsonl will show you exactly how that case's output evolved as you changed the prompt — which is the record that turns prompting from folklore into something you can review.

Step 7: Run the whole thing end to end (5 min)

Wire the pieces into one main and confirm the loop closes: prompt in, scored results out, cost reported, log written.

if __name__ == "__main__":
    print("=== V1 baseline ===")
    evaluate(SYSTEM_V1)
    print("\nLog written to", LOG.resolve())

python workspace.py

You get pass/fail per case, a score, and a prompt_log.jsonl growing by one line per call. That's the workspace. It's small on purpose — every part earns its place, and you'll extend it in later parts rather than replace it.

What you just did

You replaced eyeballing-a-chat-window with a measured loop: fixed cases → run → score → compare → cost → log. The change cycle is now under a minute, and — more important — every change is judged against a constant instead of a single lucky input. That's the difference between prompt engineering and prompt guessing, and it's the foundation the rest of this path assumes you have.

What each part of the workspace buys you

Workspace part	What it buys you	Symptom when you skip it
EVAL_CASES — fixed test cases	The same 5–10 inputs every run, so a change is measured against a constant	You tune on the easy ticket, ship it, and never see the hard ones regress
evaluate() — a runner that scores	"Improved 7 of 10 and broke 1" instead of "it looks better"	Failures stay invisible until someone downstream finds them
compare() — side by side	"On no_issue, A invents a problem and B says there's nothing to summarize"	You ship on a hunch about which version reads better
cost() — token and dollar estimate	The number before the invoice	A 500-case regression suite with long inputs becomes real money quietly
log_run() → prompt_log.jsonl	grep no_issue prompt_log.jsonl shows how a case evolved	You can't remember which prompt version you shipped, and rediscover the same fix

You're done when

python workspace.py prints one PASS/FAIL line per case and ends with 2/3 passed, with no_issue among the failures — if every case passes on a one-line system prompt, your cases are too easy to be measuring anything.
prompt_log.jsonl exists next to the script and gains one line per call, each carrying tokens_in, tokens_out and a cost you can add up.
compare(SYSTEM_V1, SYSTEM_V1) prints an A block and a B block for all three cases — that's the plumbing you'll point at two genuinely different prompts in Part 2.

Troubleshooting

AuthenticationError / 401. The ANTHROPIC_API_KEY variable isn't set in this shell. Environment variables don't survive a new terminal — re-run the export line, or use a .env file with python-dotenv. Confirm with echo $ANTHROPIC_API_KEY.

NotFoundError / model not found. The MODEL string doesn't match a model your key can access. Check your provider's model list and update the one MODEL = line.

RateLimitError / 429. You're calling too fast, common when an eval loops over many cases. Add a short time.sleep(0.5) between calls, or catch the error and retry with backoff.

IndexError on content[0]. The response came back with no text block — usually because max_tokens was so low the model was cut off before writing anything, or the model returned only a tool/other block. Raise max_tokens and print the raw resp to see what came back.

The bill is higher than expected. You're probably logging huge inputs or running the full eval more often than you think. Check prompt_log.jsonl — the tokens_in column tells you which cases are expensive.

Common mistakes

No fixed cases. The one that guarantees you're guessing. Everything else is optional; this isn't.
Only happy-path cases. Without a should-produce-restraint case like no_issue, you never catch the model inventing things to be helpful.
Cost rates buried in the code. Put them in two named constants so updating them is a ten-second job, not an archaeology dig.
Skipping the log. The first time you can't remember which prompt version you shipped, you'll wish you had it.

Continue the Practical Prompt Engineering path

Next — Part 2: Write Your First System Prompt — we fix the two failures this workspace just exposed.

For the reasoning behind this step-by-step guide, read Prompting 101: How to Talk to AI.