Multi-Turn Conversations — Practical Prompt Engineering, Part 7

Shawn West
Jul 8
11 min read

Updated: Aug 18

Practical Prompt Engineering · Part 7

Every turn you add to a conversation, you resend every turn before it — the detail the tidy chat-loop diagrams leave out. By turn 40 of a support session you aren't paying to process one new message; you're reprocessing the entire transcript, again, on every reply. Input tokens grow linearly with turns, so cost grows quadratically over the life of a conversation. And there's a hard wall past the money: the transcript eventually stops fitting in the model's context window, and the call fails with an error instead of a reply.

Context management is the work that keeps neither of those on a customer. The summarizer from Part 1 was single-shot — one ticket in, one summary out. Now it becomes a live support agent that talks back and forth with a named customer across a long session, and across sessions when she returns next week. Same support world, one new problem: history that grows without bound.

Before you start

workspace.py from Part 1 sitting next to your new conversation.py. Part 7 imports run, MODEL and client from it directly — without Part 1's harness, nothing here runs.
Python 3.10+ with the anthropic SDK installed in your virtual environment, and ANTHROPIC_API_KEY set in your environment rather than in the file.
A key that can reach the model you're using and the count_tokens endpoint. Budget for real spend: the hybrid strategy adds a summarizer call, and persistence adds a facts-extraction call, on top of every reply.
A writable working directory. The Conversation class creates a conversations/ folder and writes one JSON file per user_id.
Your provider's current context-window number to hand. CONTEXT_WINDOW is a constant you will bump, not a fact.
Block an uninterrupted hour. Most of it is driving a session long enough to trip the trimmer, and running it twice to prove persistence.

What you'll build

An agent that holds a running conversation, watches its own token count against a real window, and trims itself before it gets expensive or overflows — then persists that state so a returning customer isn't a stranger. It builds directly on workspace.py from Part 1 (run, MODEL, and the Anthropic client), so keep that file next to this one.

The basic pattern: append and resend

A conversation is just a list of {"role", "content"} messages. Each turn you append the user's message, call the model, and append the reply:

# conversation.py — builds on workspace.py from Part 1
from workspace import run, MODEL, client   # run(system, user) -> (text, usage)

def chat(system, history, user_message):
    """One turn. Returns (reply_text, new_history)."""
    messages = history + [{"role": "user", "content": user_message}]
    resp = client.messages.create(
        model=MODEL,                 # "claude-sonnet-5" — one constant, swappable
        max_tokens=1024,
        system=system,
        messages=messages,
    )
    reply = resp.content[0].text
    return reply, messages + [{"role": "assistant", "content": reply}]

That's the whole mechanism, and it's fine for ten turns. The problem is what history does over a hundred: it only ever gets longer, and you pay for all of it every time.

Counting tokens against a real window

You can't manage what you don't measure, so count before every call. claude-sonnet-5 today exposes a 200,000-token context window (some tiers offer ~1M). Treat that number as a moving target — windows vary by model and have only grown over the past two years — which is why it lives in one constant you can bump, not scattered through your logic.

Don't estimate with len(text) / 4; it drifts badly on real support text full of order numbers and URLs, and the drift runs in the direction that blows a budget rather than pads it. Use the provider's own counter — Anthropic ships a count_tokens endpoint that runs Claude's tokenizer:

CONTEXT_WINDOW = 200_000     # claude-sonnet-5 today; varies by model, keeps growing
BUDGET = 0.75                # trim well before the wall — for cost, not just capacity

def count_tokens(system, messages):
    r = client.messages.count_tokens(model=MODEL, system=system, messages=messages)
    return r.input_tokens

A note if you come from the OpenAI world: tiktoken counts offline, but its encodings are OpenAI's, not Claude's — point it at a Claude model and the number is confidently wrong. Match the tokenizer to the model, or just call the endpoint above.

Here's the part worth internalizing: I trim at 75% of the window, not 99%. Cost hurts before capacity does. A 150k-token transcript that technically fits still charges you for 150k input tokens on every remaining turn. Trimming early is a bill decision as much as a capacity one.

Strategy 1: truncate the oldest turns

The bluntest fix. When you're over budget, drop the oldest user/assistant pair until you're under:

def truncate_oldest(system, history, target):
    while len(history) > 2 and count_tokens(system, history) > target:
        history = history[2:]        # drop the oldest exchange
    return history

Cheap, instant, no extra model call. And it will burn you. In our support session, turns 1–3 are where the customer said "my order is #12345" and "ship it to the new address." Truncation throws exactly that away, and three turns later the agent asks for an order number the customer already gave — the classic "it forgot" complaint. Reach for this only when the early turns are genuinely disposable, never when identifying facts live up front.

Strategy 2: summarize the middle

Instead of deleting old turns, compress them. Take the oldest chunk, ask the model to boil it down, and replace many turns with a few lines:

def summarize(old_messages):
    transcript = "\n".join(f"{m['role']}: {m['content']}" for m in old_messages)
    instruction = (
        "Compress this support conversation into 4–6 bullet points. Preserve: the "
        "customer's identity, any order numbers, what they asked for, what the agent "
        "promised, and anything still unresolved. Drop pleasantries. Invent nothing.\n\n"
        + transcript
    )
    summary, _ = run("You are a precise conversation summarizer.", instruction)
    return summary

Better memory than truncation, but it isn't free: it costs an extra model call every time it fires, and it's lossy in ways you don't control — more on that below, since the strategy I actually ship leans on it.

Strategy 3 (my default): the hybrid

Neither pure strategy is right, because a conversation has three zones that deserve different treatment. The last few turns are what the model is actively reasoning about — those must survive verbatim, or replies get vague and repetitive. The middle is context you still want but can afford to compress. The very oldest greetings are usually noise. So: keep recent turns exactly, summarize the middle, let the summary absorb the oldest. That's the hybrid, and it's what I reach for by default on any long-running conversation.

One correctness trap first, because the naive version is subtly broken. The obvious move is to inject the summary as a {"role": "system", ...} message inside the messages list. Anthropic rejects that — system is a top-level parameter, not a valid role inside messages. The fix is to fold the summary in as a synthetic user/assistant exchange, which also keeps roles strictly alternating (Claude requires that):

KEEP_RECENT = 6   # last 3 exchanges (user+assistant) survive verbatim

def manage_context(system, history, target):
    if count_tokens(system, history) <= target:
        return history                      # nothing to do

    recent = history[-KEEP_RECENT:]         # active reasoning zone — untouched
    middle = history[:-KEEP_RECENT]         # everything older — compress
    summary = summarize(middle)

    stitched = [
        {"role": "user",
         "content": f"[Earlier in this conversation]\n{summary}"},
        {"role": "assistant",
         "content": "Got it — I have that context and will keep it in mind."},
        *recent,
    ]
    return stitched

Why this shape holds up: the summary carries the durable facts (order #12345, the address change, the promised refund) forward at a fraction of the tokens, while the last three exchanges stay word-for-word so the model's short-term thread is intact. KEEP_RECENT is the dial that matters. Set it too low — say 2 — and you summarize away the turn the customer is literally responding to, and the agent answers a question that's already moved on. Set it too high and you've barely saved tokens. Six (three exchanges) is a sound starting point for support; bump it for reasoning-heavy chats where the recent thread carries more weight.

Be honest about what summarization loses. A summary is a lossy compression chosen by a model, and it discards exactly what it wasn't told to keep. Verbatim quotes go first — "it's been two weeks and this is the third time I've contacted you" flattens to "customer is frustrated," and the escalation signal is gone. Precise values drift: order #12345 comes back as "the customer's order" if your summarizer prompt didn't pin identifiers, which is why the prompt above names them. Timestamps and ordering blur — "before" and "after" can invert. And a summary can quietly invent: told to be concise, a model smooths a hedged "I'll check if a refund is possible" into "agent promised a refund," and your transcript now records a commitment that was never made. That last failure is the dangerous one, because nothing downstream flags it. The mitigation is a summarizer prompt that names what to preserve and says invent nothing — but even then, treat the summary as a good-enough sketch, never a faithful record.

That trade is why the hybrid wins: the turns most likely to be misquoted by a summary — the recent ones — are the ones it never touches.

Persisting state across sessions

Everything so far dies when the process exits, so a returning customer re-explains her order every visit. Persist two things between sessions: the history (trimmed) and a small set of extracted user facts that ride in the system prompt, so the agent knows her without re-reading the whole transcript.

import json, pathlib

STORE = pathlib.Path("conversations"); STORE.mkdir(exist_ok=True)
BASE_SYSTEM = ("You are a support agent for an online store. Be concise and accurate. "
               "Never invent order details or make promises you can't verify.")

class Conversation:
    def __init__(self, user_id):
        self.user_id = user_id
        self.path = STORE / f"{user_id}.json"
        if self.path.exists():
            data = json.loads(self.path.read_text(encoding="utf-8"))
            self.history, self.facts = data["history"], data["facts"]
        else:
            self.history, self.facts = [], {}

    def _system(self):
        if not self.facts:
            return BASE_SYSTEM
        facts = "\n".join(f"- {k}: {v}" for k, v in self.facts.items())
        return f"{BASE_SYSTEM}\n\nKnown facts about this customer:\n{facts}"

    def send(self, message):
        system = self._system()
        target = int(CONTEXT_WINDOW * BUDGET)
        probe = self.history + [{"role": "user", "content": message}]
        if count_tokens(system, probe) > target:
            print(f"  [context] {count_tokens(system, probe):,} tok > {target:,} — managing")
            self.history = manage_context(system, self.history, target)
        reply, self.history = chat(system, self.history, message)
        self._learn(message, reply)
        self._save()
        return reply

    def _learn(self, message, reply):
        prompt = ("Extract durable facts about the customer as strict JSON "
                  '(keys like name, order_id, plan, address). Return {} if none.\n\n'
                  f"Customer: {message}\nAgent: {reply}")
        raw, _ = run("You extract stable user facts as strict JSON. Output JSON only.", prompt)
        try:
            self.facts.update(json.loads(raw))
        except json.JSONDecodeError:
            pass   # a non-JSON reply just means "nothing durable this turn"

    def _save(self):
        self.path.write_text(
            json.dumps({"history": self.history, "facts": self.facts}),
            encoding="utf-8")

The facts extractor is deliberately narrow: it pulls stable identity (name, order_id, plan tier), not the content of every turn. That keeps the system prompt to a handful of lines instead of a growing pile — the difference between "remembers who you are" and "replays your whole history at you."

Running it end to end

Drive a session long enough to trip the trimmer. To see management fire without a 200k-token transcript, drop BUDGET low for the demo — which mirrors reality, where you cap context for cost well before the window is the constraint:

if __name__ == "__main__":
    BUDGET = 0.02   # ~4,000-token cap, so trimming triggers on a short demo
    convo = Conversation("priya-88")
    for msg in [
        "Hi, order #12345 still hasn't arrived and it's been two weeks.",
        "Can you ship the replacement to my new address, 55 Elm St?",
        "How long will the replacement take?",
        # ...many turns of back-and-forth...
        "Actually, can you just refund it instead?",
    ]:
        print("USER:", msg)
        print("AGENT:", convo.send(msg), "\n")
    print("Learned facts:", convo.facts)

A realistic trace once the transcript crosses the cap:

  [context] 4,180 tok > 4,000 — managing
  [context] summarized 11 middle turns -> 96 tokens; kept last 6 verbatim
AGENT: I can refund order #12345 to your original payment method...
Learned facts: {'order_id': '12345', 'address': '55 Elm St'}

Run it twice. The second run loads priya-88.json, and the agent already knows the order number and address — no re-explaining. That's the payoff: cheap, under the window, and remembers the customer across visits.

Choosing a context strategy

Strategy	What it costs	When it breaks
Append and resend, unmanaged	Every prior turn re-billed on every reply; cost grows quadratically over the conversation	The transcript stops fitting the window and the call fails with an error instead of a reply
Truncate the oldest turns	Nothing extra — no model call, instant	Identity lives in the oldest turns; drop the first few and the agent asks for an order number the customer already gave
Summarize the middle	An extra model call every time it fires	Lossy in ways you don't control — quotes flatten, identifiers drift, and a hedged "I'll check if a refund is possible" can harden into "agent promised a refund"
Hybrid: keep recent, summarize middle	One summarizer call, plus KEEP_RECENT turns held at full price	KEEP_RECENT too low and you summarize away the turn the customer is answering; too high and you've barely saved tokens
Persisted facts in the system prompt	A facts-extraction call per turn	Only holds for stable identity — name, order id, plan, address. Persist the whole transcript as "memory" and you've just moved the cost problem

You're done when

You run the demo with a deliberately low budget and the console prints a [context] line before the reply — the token count, the target it exceeded, and how many middle turns were summarized against how many were kept verbatim.
You run the same script a second time, it loads the saved conversation file, and the agent answers using the order number and address without the customer restating either.
You log count_tokens before every call across a long session and the number rises and then drops when manage_context fires, instead of climbing turn over turn forever.

Troubleshooting

The model "forgot" something from earlier. Check whether the fact was ever in the window — if truncation or an over-aggressive summary dropped it, the model can't recall what it never saw. This is usually a context-management bug, not a model failure. Raise KEEP_RECENT, or promote the fact to user_facts so it rides in the system prompt permanently instead of living in trimmable history.

The bill is ballooning. Almost always unbounded history — you're resending a 60-turn transcript on every reply. Log count_tokens before each call; if it climbs turn over turn and never drops, manage_context isn't firing (check BUDGET) or you're persisting the untrimmed history. Save the trimmed self.history, not the raw one.

Summaries are dropping or inventing facts. Tighten the summarizer prompt to name what must survive (order numbers, promises, unresolved items) and to say invent nothing. If it still fabricates commitments, shrink the chunk you summarize per call — a summary of 30 turns loses more than three summaries of 10.

count_tokens and the actual bill disagree. You're likely counting with the wrong tokenizer (e.g. tiktoken against a Claude model). Use the provider's endpoint for the model you're calling.

Common mistakes

Injecting the summary as a role: "system" message inside messages. Anthropic rejects it; system is a top-level param. Stitch summaries in as a synthetic user/assistant pair.
Truncating when identity lives in the oldest turns. The order number stated in turn 1 is the first casualty. Extract it to user_facts before you trim.
Trimming at 99% of the window. By then you've overpaid for dozens of turns. Trim at a budget you set for cost, not at the capacity wall.
Persisting the whole transcript as "memory." Loading a 200-turn history every session just moves the cost problem; a few extracted facts is what "remembering" should cost.
Trusting the summary as a record. If a promise or a number must be exact, keep it verbatim or store it as a fact — don't let a summarizer paraphrase it.