Context Engineering vs Prompt Engineering: How to Tell Which One Is Broken
Prompt engineering changes the instructions; context engineering changes what the model is given to work with. When the second is at fault, no amount of rewording the first will fix it. The way to tell them apart is to replay the exact context the model saw and change one layer at a time.
The example in this article is a composite scenario built from patterns common to customer-support assistants. It is not a single real organisation. Its figures come from a seeded simulation published with the article, in which the model is a labelled stand-in, and are illustrative.
An online retailer's support assistant answered refund questions from the company's policy documents. On 1 September the refund window for electronics changed from 30 days to 14. Two weeks later, customers were still being told they had 30 days, and the escalations were landing on the support lead's desk.
The team treated it as a prompt problem, because the prompt was the thing they owned. Version 2 added "Be accurate. Do not guess." Version 3 added "Read every policy document carefully before answering." Version 4 added "If policies conflict, use the one with the latest effective date." Each version passed the assistant's evaluation set with no errors. In production, replayed against 500 logged conversations, every one of the four versions gave the wrong refund window 315 times.
The prompt was never the problem. The old policy document had never been retired from the knowledge base, retrieval ranked it above the new one more often than not, and the documents reached the model without their effective dates. Nothing in the prompt could fix that, because the information the fix needed wasn't in the context.
What each one changes
The two terms describe different layers of the same model call. Anthropic's engineering team describes prompt engineering as "writing and organizing LLM instructions for optimal outcomes", and context engineering as "the set of strategies for curating and maintaining the optimal set of tokens (information) during LLM inference": the instructions plus tools, external data, message history and everything else that enters the window (Anthropic, 2025). It calls context engineering the natural progression of prompt engineering, not a replacement.
The useful difference for someone debugging is not the definition but the operating profile.
Prompt engineering | Context engineering | |
What it changes | The instructions: role, task, rules, examples, output format | Everything else in the window: retrieved documents, tool results, history, memory, metadata |
When it is decided | At design time, then versioned | On every call, by code at runtime |
Who usually owns it | Whoever writes the prompt | Whoever builds retrieval, tools and memory, often a different person |
Typical failure | The model misreads what it was asked to do | The model faithfully uses wrong, stale, missing or conflicting information |
How the failure shows up | Consistently, on similar inputs | On some inputs and not others, depending on what was assembled |
The artefact | A prompt spec, under version control | A context manifest per call, in the trace |
How you test it | Fixed inputs, fixed context, vary the instructions | Logged contexts, replayed, with the assembly changed |
The two layers interact, and that is where most confusion comes from. An instruction can only act on information the context provides. "Use the latest effective date" is a sound instruction, and it does nothing if the documents arrive without dates. The prompt side is covered in depth in The Prompt Spec; the context side in Context Engineering: A Practitioner's Guide.
The test: for your last three fixes to an LLM feature, write down which layer each one changed. If all three changed the prompt, check whether the failures they addressed were consistent or input-dependent.
Why rewording the prompt is the reflex
The prompt is the visible artefact. It sits in the repository, it has an owner, and a change to it ships in minutes. Context is assembled by retrieval code, a knowledge base and a conversation store that may belong to three different people, and the assembled result usually isn't logged at all.
The evaluation set reinforces the reflex. Most eval sets are written by people who know the right answer, so each case pastes in the correct document. That makes the eval set a test of the prompt alone: the context is always right, so the only thing that can fail is the instructions. The refund assistant's eval set had 50 questions, each with the current policy pasted in. It could never have contained the old document, so it could never have caught the failure, and every prompt version passed it.
There is also an honest reason. Adding a sentence to the prompt sometimes does fix a context problem, by restating the fact the context should have carried: "The electronics refund window is 14 days." That works until the next policy change, and the prompt has quietly become a second knowledge base that nobody maintains.
The test: open your eval set and check where each case's context comes from. If a person pasted it in, the set tests your prompt and nothing else. Evaluating AI Agents covers building cases from production instead.
How to tell which one is broken: replay, then change one layer
The triage is the same experiment you would run on any system with two suspects: hold one constant and vary the other. It needs one thing many teams don't have, which is the exact context each production call received.
Replay that context against the current prompt, and you should see the production failure. Then run two variations. Keep the context fixed and change the prompt: if the failure goes away, the instructions were the problem. Keep the prompt fixed and correct the context, by retiring a document, adding metadata, or changing what retrieval returns: if the failure goes away, it was the context.
Why context failures are so common has a mechanism behind it. Models don't use their context evenly. Researchers found performance is highest when relevant information sits at the beginning or end of the input, and drops when it sits in the middle (Liu et al., 2024). A 2025 study of 18 models found that performance degrades as input length increases, "often in surprising and non-uniform ways", and that "even a single distractor", a passage that looks relevant but is wrong, reduces performance compared with the relevant passage alone (Hong, Troynikov and Huber, 2025). A superseded policy is exactly that kind of distractor: it is on topic, confidently written and wrong.
The test: pick one wrong answer from last week and try to retrieve the exact context the model received for it. If you can't, triage is guesswork, and The AI Agent Observability Checklist lists what to record.
The refund assistant, triaged
The published script replays the 500 logged contexts and varies one thing at a time. The model in it is a stand-in, labelled as such: a deterministic policy that answers from the first document stating a refund window, and prefers the most recent policy only if told to and only if the documents carry dates. That encodes the behaviour the incident showed, so the output can be reproduced exactly; a real model would vary from run to run, so with one you replay each case several times.
== the team's eval set (50 questions, current policy pasted in)
prompt v1 wrong 0 / 50
prompt v2 wrong 0 / 50
prompt v3 wrong 0 / 50
prompt v4 wrong 0 / 50
== replayed production contexts (500 logged runs, as assembled)
prompt v1 wrong 315 / 500
prompt v2 wrong 315 / 500
prompt v3 wrong 315 / 500
prompt v4 wrong 315 / 500
== same logged runs, one change at a time
prompt v1, superseded policy dropped wrong 0 / 500
prompt v4, effective dates carried wrong 0 / 500
prompt v1, effective dates carried wrong 315 / 500
old policy ranked first in logged contexts 315 / 500
(Full contents of output.txt.)
Three things in that output are worth reading slowly.
First, the eval set and production disagree completely, and they disagree because of where their contexts came from, not because of the prompt. Every version is perfect on pasted-in context and wrong 315 times on assembled context.
Second, the wrong answers line up exactly with the cases where the old policy ranked first: 315 of 500. That count, from the logged contexts alone, points at retrieval before anyone touches a prompt.
Third, the two layers interact. Version 4's instruction to prefer the latest effective date fixes every case, but only once the context carries the dates. Carrying the dates without the instruction fixes nothing. Dropping the superseded policy fixes everything with the original prompt. The cheapest durable fix was a context change: retire superseded policies from the knowledge base, and filter retrieval to policies in effect. The team kept version 4's instruction and the dates as a second line of defence.
The discovery question that would have prevented it is a data question, asked when the assistant was scoped: when a policy changes, what happens to the old document? The answer was "nothing", and nobody had asked.
The test: count how many documents in your knowledge base have a newer version that supersedes them. Each one is a distractor waiting to be retrieved.
Failure signatures: which layer to suspect first
The replay is the proof. These signatures tell you where to look first.
What you see | Suspect first | Confirm by |
Wrong on some inputs, right on similar ones | Context | Compare the assembled contexts of a right and a wrong case |
Wrong consistently, even with the right document pasted in | Prompt | Run the eval case with the context fixed; vary the instructions |
Output format or tone is wrong | Prompt | The output contract in the instructions |
Right early in a conversation, wrong later | Context: history, truncation or compaction | Replay the late turn with the history trimmed |
Fixed by adding a fact to the prompt | Context, in disguise | Ask where that fact should have come from |
Wrong after a knowledge-base or tool change, with no prompt change | Context | Diff the assembled contexts before and after |
Ignores a rule only when a retrieved document contradicts it | Both: instruction priority and untrusted content | Replay with the document marked as reference material |
The last row is where the layers overlap most. Retrieved text that contains instructions, intended or not, competes with your prompt. Separating and labelling external content is a context-assembly job and a prompt job at once, and it is the mitigation OWASP gives for indirect prompt injection: "separate and clearly denote untrusted content to limit its influence" (OWASP, 2025).
The triage method
The method takes an afternoon the first time, and minutes once the pieces are in place. It needs the prompt owner and whoever owns retrieval and memory.
Log the context manifest for every call. Prompt version, the documents and tool results included (with their IDs and versions), and the history length. Done when any production answer can be traced to the exact inputs the model saw.
Build a replay set from production, not from memory. Take the failing cases plus a sample of passing ones, with their logged contexts. Done when replaying the current prompt reproduces the failure.
Change one layer at a time. Prompt changes on fixed contexts; context changes on a fixed prompt. Done when one change, on its own, fixes the replayed failures, or you have shown it takes both.
Fix it in the layer it lives in. Retire or version the stale document, add the missing metadata, change retrieval; or change the instruction. Don't restate a changing fact in the prompt. Done when the fix is in the system that owns the information.
Add the case to both test sets. A replay case with its real context, and a prompt regression case. Done when the failure would now fail a build, as described in Prompt Regression Testing in CI.
Tradeoffs and exceptions
Sometimes the prompt really is the problem. A model that ignores an output format, misreads the task or applies a rule inconsistently with the right information in front of it needs instruction work, and the Prompt Spec is the place to do it. The replay tells you so: if a prompt change fixes the failure on the fixed, logged context, believe it.
Restating facts in the prompt is reasonable for facts that change rarely and matter on every call: the company's name, the product line, a legal disclaimer. It is a trap for facts with a lifecycle, such as prices, policies, limits and dates, because the prompt has no owner for their accuracy.
Logging full contexts has a cost. Retrieved documents and conversation history can contain personal data, so the manifest should usually record document IDs and versions plus a hash of the assembled text, with full content kept only in a restricted, short-retention store. The observability checklist covers the defaults.
Finally, more context is not a safe default. The Chroma results are a reminder that adding documents to be safe adds distractors. The context fix is usually to remove or filter, not to add.
How the triage goes wrong
The replay uses today's context, not the logged one. The knowledge base has changed since the failure, so the replay passes and the bug looks fixed. Replay what was logged.
One run per case. A real model's answer varies; a single replay is a sample. Run each case several times and compare rates.
Both layers changed at once. A prompt edit and a retrieval change ship together, the failure goes away, and nobody knows which fixed it. The next regression will be just as hard.
The fix lands in the prompt because it's faster. The incident closes, the stale document stays, and the next policy change repeats the incident.
What to do next
Take the most recent wrong answer your LLM feature gave in production. Try to retrieve the exact context the model received. If you can, run step 3 of the method on it this week. If you can't, that is the first fix: start logging the context manifest, and treat every prompt change until then as a guess.
Final takeaway
Prompt engineering and context engineering are two layers of the same call, and they fail differently: instructions fail consistently, context fails depending on what was assembled. Replay the logged context and change one layer at a time before you rewrite anything. When the information the model needs isn't in its context, no instruction can supply it.
Sources
Anthropic Applied AI team, Effective context engineering for AI agents, Anthropic Engineering, 29 September 2025. Definitions of prompt and context engineering; context engineering as the natural progression of prompt engineering; what context includes.
Hong, K., Troynikov, A. and Huber, J., Context Rot: How Increasing Input Tokens Impacts LLM Performance, Chroma, 14 July 2025. 18 models; performance degrades with input length; a single distractor reduces performance.
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F. and Liang, P., Lost in the Middle: How Language Models Use Long Contexts, Transactions of the Association for Computational Linguistics, vol. 12, 2024 (arXiv 2307.03172, 2023). Performance highest when relevant information is at the beginning or end.
OWASP GenAI Security Project, LLM01:2025 Prompt Injection, 2025. Indirect prompt injection; separate and clearly denote untrusted content.
Frequently Asked Questions
What is the difference between context engineering and prompt engineering?
Prompt engineering is writing and organising the instructions a model follows. Context engineering is deciding everything else the model is given on each call: retrieved documents, tool results, conversation history, memory and metadata. Instructions are mostly fixed at design time; context is assembled fresh on every call.
Is context engineering replacing prompt engineering?
No. Anthropic describes context engineering as the natural progression of prompt engineering, and the two depend on each other. An instruction can only act on information the context provides, and well-assembled context still needs clear instructions about what to do with it.
How do I know whether a wrong answer is a prompt problem or a context problem?
Replay the exact context the model received in production. Hold the context fixed and change the prompt, then hold the prompt fixed and correct the context. Whichever change fixes the replayed cases tells you the layer. If your eval set pastes in the right context, it can only ever find prompt problems.
Part of the Agent Engineering curriculum: eight paths from prompts and context to testing, evaluation and governance.
