LLM Application Patterns: From Simple Completions to Reasoning Systems
Updated: 15 hours ago
The standard advice for LLM applications is to start with the simplest pattern and add complexity only when measurement proves it's needed. It's good advice, and teams ignore it constantly, because it leaves out the method. Which measurement? Proves what? Needed for which failure?
Without those answers, the choice of pattern is made by whatever the team read about last. A team that has just seen an agent demo builds an agent. A team that has heard RAG reduces hallucination adds a vector database to a problem that never lacked information.
This post gives the method. Each pattern fixes a specific kind of failure, so the order is: build an evaluation set, measure the simplest pattern, sort its failures by type, and let the most common type choose the next step. The earlier posts in this path covered infrastructure, evaluation and safety. This one is about structure: how the application, the model and external systems fit together.
The running example
(Composite.) A platform team wants an assistant that triages CI test failures. For each failed run, it should classify the failure as a product bug, a flaky test, an infrastructure problem or a test that needs updating, and suggest an owning team. Today, an engineer on rotation does this by hand for about 40 failures a day.
The team's first move, before choosing any pattern, is the one most teams skip: they pull 200 historical failures that engineers have already triaged and label each with the correct category and owner. That labelled set is the measurement the advice refers to. Every pattern decision below is a comparison on those 200 cases.
The test: do you have a labelled set of real inputs with known correct outputs? If not, you can't follow "add complexity when measurement proves it's needed", because you have nothing to measure. Building one is the first engineering task, not a later one. A test strategy for LLM features covers how to build and maintain it.
Pattern 1: a single completion, and how to tell it isn't enough
What it is: one prompt in, one response out. A template, the input, and the model's answer.
Failure log + test name → Prompt template → LLM → Category + owner
For many tasks this is the whole system: classification, summarisation, extraction, rewriting. It's cheap, fast, predictable and easy to debug. The question is never whether it's elegant enough. It's whether it's accurate enough on your eval set.
In the example: a single completion with a careful prompt gets 142 of 200 cases right (71%). That number alone doesn't tell you what to build next. The errors do. The team reads all 58 failures and sorts them:
Error type | Count | What was missing | Pattern it points to |
Didn't know the test was on the known-flaky list | 24 | Facts the team has, the model doesn't | Retrieval |
Didn't know about an infrastructure incident that day | 11 | Facts the team has, the model doesn't | Retrieval |
Couldn't tell whether the same test was failing on main | 14 | Live state from another system | Tool use |
Misread a long, multi-cause log | 6 | Reasoning over a complex input | Decomposition |
Labelling disagreement in the eval set | 3 | Nothing: the label was wrong | Fix the eval set |
This table is the decision. 35 errors come from missing facts, 14 from missing live state, 6 from reasoning. An agent would address none of these in particular.
The test: after measuring a single completion, read every failure (or a random 50 if there are hundreds) and sort them by what was missing. If you can't sort them, the eval set or the task definition is too vague, and that's the problem to fix next. Discovery for AI features covers scoping the task before the prompt.
Pattern 2: retrieval, when the model lacks your facts
What it is: fetch the relevant information and put it in the prompt before the model answers. The original retrieval-augmented generation paper (Lewis and colleagues, 2020) paired a retriever with a generator for knowledge-intensive tasks. In practice, retrieval means anything that gets the right facts into the context.
Failure → Look up known-flaky list + today's incidents → Augmented prompt → LLM → Category + owner
The retrieval doesn't have to involve embeddings or a vector database. In the example, the known-flaky list has 300 entries and the incident log for one day has a handful, so the team looks them up by test name and date and puts the matches in the prompt. A vector search would add a component to tune and a new way to miss.
Measure retrieval separately from generation. For each eval case, record whether the facts needed to answer were actually in the prompt. If the right fact was retrieved and the answer was still wrong, that's a generation problem. If the fact wasn't retrieved, no prompt change will help.
In the example: adding the two look-ups brings accuracy to 172 of 200 (86%). Of the 35 missing-fact errors, 30 are fixed. The 5 that remain were tests renamed since they joined the flaky list, which is a data problem, fixed by keying the list on test ID instead of name.
The test: for each wrong answer, check whether the information needed was in the prompt. If it usually was, retrieval isn't your bottleneck. If it usually wasn't, improve the look-up before touching the prompt or the model.
Pattern 3: decomposition and reasoning, when the input is complex
What it is: break a hard task into steps. The early research showed large gains on reasoning tasks from prompting models to work through intermediate steps (Wei and colleagues, 2022), and from sampling several reasoning paths and taking the majority answer (self-consistency, Wang and colleagues, 2022).
Two things have changed since. First, many current models are reasoning models that work through problems internally, and their vendors now advise against asking them to "think step by step": OpenAI's guidance for its reasoning models says such prompts are unnecessary and can sometimes hinder performance. Second, the decomposition worth doing is usually structural: your application splits the task, rather than the prompt asking the model to.
# Structural decomposition for a long, multi-cause log
sections = split_log_by_stage(log) # setup, build, test, teardown
findings = [summarise_stage(s) for s in sections] # one short call per stage
category = classify(findings, known_issues) # final call sees a clean summary
Each step costs tokens and time, and self-consistency multiplies the cost by the number of samples. Decomposition is worth it only for the error type it fixes.
In the example: structural decomposition fixes 5 of the 6 reasoning errors, but only those; it makes no difference to the rest. Since it triples the calls per failure for about 2% of cases, the team runs it only when a log exceeds a length threshold, which is where all 6 errors were.
The test: run the eval set with and without decomposition and compare accuracy on the cases tagged as reasoning errors. If the gain is concentrated in a subset you can detect (long inputs, multiple causes), route only those cases through it.
Pattern 4: tool use, when the answer depends on live state
What it is: the model can call functions (an API, a database query, a search) and use the results. It's the right pattern when the answer depends on something that changes, or when the system needs to act.
Failure → LLM decides to query CI history → get_test_history(test_id, branch="main", hours=24)
→ Result → LLM → Category + owner
Three design rules matter more than the framework you use.
Describe tools precisely. The model chooses tools from their descriptions. Say what the tool returns and when to use it, with an example.
Validate every call before running it. The model writes the arguments. Check them against a schema, and reject anything outside the allowed range.
Grant the least privilege that works. A triage assistant needs read access to CI history. It doesn't need to rerun pipelines, and certainly not to deploy.
Log tool calls as their own measurement: did the model call the right tool, with valid arguments, and did it use the result correctly? These fail independently of the final answer.
In the example: a single read-only tool for test history fixes 13 of the 14 live-state errors, bringing accuracy, with the routed decomposition, to 190 of 200 (95%). Most of the remaining ten are the renamed tests and the three bad labels, both fixes to data rather than to the pattern.
The test: for every tool, count the calls in your eval run that were unnecessary, used the wrong tool or had invalid arguments. If the wrong-tool rate is above a few percent, fix the descriptions before adding more tools.
Pattern 5: agents, and the arithmetic of compounding
What it is: the model runs in a loop, deciding what to do next, acting, observing the result, and deciding when it's finished. Anthropic's guide to building agents draws a useful line here. Workflows are LLMs and tools orchestrated through predefined code paths. Agents direct their own process and tool use. Everything above is a workflow. Anthropic recommends agents for open-ended problems where it's hard or impossible to predict the number of steps, and warns that their autonomy means higher costs and the potential for compounding errors.
The compounding is worth doing as arithmetic. If each step succeeds with probability p, and errors are independent and never corrected, a task of n steps succeeds with probability pⁿ:
Per-step success | 5 steps | 10 steps | 20 steps |
99% | 95% | 90% | 82% |
95% | 77% | 60% | 36% |
90% | 59% | 35% | 12% |
Real agents can notice and recover from some errors, so the table is a pessimistic bound, not a forecast. But it shows why a demo that works, which is usually a short path, says little about a long one, and why the checks inside the loop matter more than the model.
In the example: someone proposes extending the assistant into an agent that reruns suspected flaky tests, files tickets and pings the owning team. But the path is predictable (classify, check history, rerun if flaky, file a ticket if not), so it's a workflow, not an agent. Rerunning is safe to automate. Filing tickets and paging people aren't easily undone, so the workflow drafts them and an engineer confirms. That decision is the autonomy question that What Is AI Agent Management treats as something earned with evidence, not set by default.
If you do build an agent, keep the guardrails from the start: a hard limit on iterations, a budget per task, human confirmation for anything irreversible, and a narrow tool set. Test it in a sandbox against the same kind of labelled set, and measure the whole trajectory, not just the final answer.
The test: write down the steps your task needs. If you can write them in order, build a workflow. Choose an agent only when the number and order of steps genuinely depends on what each step finds, and you can afford its error rate over the longest path you expect.
Choosing the pattern: the failure decides
The failures are mostly… | Next pattern | Measure separately | Main cost |
Wrong format or instructions ignored | Better prompt, examples, structured output | Format compliance rate | Prompt work |
Facts the model doesn't have | Retrieval | Was the needed fact in the prompt? | Retrieval upkeep, longer prompts |
Reasoning errors on complex inputs | Structural decomposition, routed | Gain on the reasoning subset | Calls and latency per case |
Answers that depend on live state, or actions | Tool use | Right tool, valid arguments, result used | Security, validation, round trips |
Tasks whose steps can't be predicted | Agent, with guardrails | Whole-trajectory success, cost per task | Compounding errors, cost variance |
The direction of travel is the same as the old advice: left to right, only as far as you must. The difference is that each step now has a reason you can show someone, in the form of a count of failures it fixed.
What to do this week
Take one LLM feature you're building or running. If it has no labelled eval set, build one from 100 real inputs with known answers; that's the week's work. If it has one, run the current version, read every failure, and sort them with the table above. Then check whether the pattern you're using, or planning to add next, matches the biggest category. If it doesn't, change the plan before writing more code.
Final takeaway
"Start simple" works only with a measurement behind it. Build the labelled set first, measure the single completion, and read the failures. Missing facts call for retrieval, complex inputs for decomposition, live state for tools, and unpredictable paths, rarely, for an agent. Each pattern earns its place by the failures it fixes on your data, and the compounding arithmetic is why the last one needs the most evidence.
This completes the LLM Production Systems learning path. You've covered cost and latency tradeoffs, evaluation frameworks, safety guardrails, and application patterns. The throughline: LLM systems in production are engineering systems — they require the same discipline around reliability, cost, and observability as any other production system.
To see where your own AI feature is thinnest, the AI Readiness Assessment asks twenty-four questions across evaluation, guardrails, observability and governance.
Sources
Lewis, P. et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (2020). https://arxiv.org/abs/2005.11401
Wei, J. et al., "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" (2022). https://arxiv.org/abs/2201.11903
Wang, X. et al., "Self-Consistency Improves Chain of Thought Reasoning in Language Models" (2022). https://arxiv.org/abs/2203.11171
Yao, S. et al., "ReAct: Synergizing Reasoning and Acting in Language Models" (2022): the reason-act-observe loop behind most agent designs. https://arxiv.org/abs/2210.03629
Schluntz, E. and Zhang, B., "Building effective agents" (Anthropic, 19 December 2024): workflows versus agents, the simplest solution first, and agents' higher costs and compounding errors. https://www.anthropic.com/engineering/building-effective-agents
OpenAI, "Reasoning best practices": avoid chain-of-thought prompts with reasoning models, since they reason internally. https://developers.openai.com/api/docs/guides/reasoning-best-practices
Related reading
Keep learning. This article is part of the AI in Quality & Delivery path in the ShiftQuality Learning Center. Use AI in delivery — and evaluate it honestly — without the hype.
