Build an AI Agent, Stage 8: Retrieval That Can Say "None"
Updated: 2 hours ago
A retriever that can't return nothing will always hand the agent something to relate. Test retrieval on requests that have no answer as well as requests that have one, and calibrate a threshold on requests you then hold out, so "none" becomes a result the agent can trust.
The example in this series is a composite scenario, built from patterns common to requirements intake in regulated organisations. It is not a single real organisation, and its figures are illustrative. The code is real: every output block below was produced by running it.
The Requirements Agent's first search tool used the repository's built-in keyword search. It worked when brokers used the repository's words. They often didn't. A broker asked to "keep a 30-day window to report claims on business premises", and the search found nothing, because the requirement it duplicated, REQ-212, talked about "a first notice of loss on the commercial property line". The agent's note said no related requirements were found. The search had worked; it just couldn't see a paraphrase.
So the team switched to vector search over embeddings, returning the three nearest requirements for every request. The paraphrase misses stopped. Two weeks later a broker asked for cyber cover against ransomware attacks on small firms, a product the insurer had never offered. Nothing in the repository was related. The search returned three requirements anyway, the nearest being REQ-118, about brokers signing in "with single sign-on from their firm's identity provider". The agent proposed an overlap and quoted that sentence. The grounding check passed, because the quoted span really was in REQ-118. The analyst rejected the proposal in a few seconds.
That rejection was cheap. What it did next was not. Within a month the analysts had learned that the agent's overlaps were often nonsense, and they began rejecting them on sight, including, one week, a real conflict.
This is Stage 8 of Build an AI Agent From Scratch. Stage 5 made sure a failed search comes back as an error, never as an empty result. Stage 7 decided what the agent remembers. This stage makes the empty result mean something: a search that worked and found nothing related.
Why the usual retrieval test can't see the problem
The team had tested both search tools. They took a set of requests, each labelled with the requirement it related to, and measured how often the right one came back in the top three. Keyword search scored poorly on paraphrases; vector search scored well. By that test, the switch was a clear improvement.
The test had a blind spot built into its definition. A "found it in the top k" measure is only defined for requests that have something to find. Anthropic's write-up of contextual retrieval, for example, measures failure as "1 minus recall@20", the share of relevant chunks not retrieved in the top 20 (Anthropic, 2024). That is the right measure for what it asks. It can't score a request whose correct answer is nothing, so it can't see a retriever that never says "none".
For a requirements agent, requests with no related requirement aren't rare. They are new products, new regulations and new ideas, and they are exactly the requests where an invented relation does the most harm: it tells an analyst that something new is already covered.
The discovery move is a question for the analysts who will review the agent: which closed requests turned out to have nothing related in the repository? Their answers become the second half of the test set. In the team's case, they had never been asked, because nobody had thought of "nothing" as an answer that needed testing.
The test: look at the requests in your retrieval test set and count how many are labelled with no related record. If the answer is zero, your test can't fail a retriever that always returns something.
Three retrievers, one test with both halves
The checker for this stage, retrieval_check.py, runs three retrievers over the same labelled requests:
Keyword: BM25 over the requirement text, returning up to three requirements that share a word with the request.
Vector: similarity over meaning, always returning the three nearest requirements.
Hybrid: both signals added together, keeping only requirements above a threshold.
The vector signal in the code is a stand-in for an embedding model: a small concept map built from the analysts' glossary, so the example runs without a model and gives the same answer every time. With a real embedding model the shape of the results is the same; the numbers are not.
The labelled set is small, 18 requests against 12 requirements, and composite. It has two splits. The hybrid's threshold is set on the 8 requests in the calibration split; all three retrievers are then judged on the 10 held-out requests in the test split, 6 with a related requirement and 4 with none. The bars come from the hub's pass condition for this stage: every request with a related requirement finds it, and every request with none gets "none".
$ python retrieval_check.py
retrieval check: 10 held-out requests, 6 with a related requirement, 4 with none
keyword
found 5/6 said none 1/4 FAIL (unlabelled extras beside a hit: 7)
missed Q01 REQ-212: "Let brokers keep a 30-day window to report claims on business premises"
invented Q15 -> REQ-118: "Cyber cover for ransomware attacks on small firms"
invented Q16 -> REQ-190: "Pet insurance for working dogs"
invented Q17 -> REQ-201, REQ-119, REQ-118: "Brokers want their logo on quotes"
vector
found 6/6 said none 0/4 FAIL (unlabelled extras beside a hit: 12)
invented Q15 -> REQ-118, REQ-180, REQ-119: "Cyber cover for ransomware attacks on small firms"
invented Q16 -> REQ-190, REQ-140, REQ-180: "Pet insurance for working dogs"
invented Q17 -> REQ-201, REQ-118, REQ-119: "Brokers want their logo on quotes"
invented Q18 -> REQ-190, REQ-171, REQ-119: "Telematics discounts for young drivers"
hybrid (threshold 0.67, set on 8 other requests)
found 6/6 said none 4/4 PASS (unlabelled extras beside a hit: 3)
meets both bars: hybrid
(From stage8-output.txt. The requests and requirements are composite; the scores are computed by the script.)
Read it by retriever. Keyword search misses the broker's paraphrase, Q01, and also invents relations whenever a request shares an incidental word: "firms" with REQ-118, "working" with the complaints requirement's "working days". Vector search finds every related requirement and never says "none": all four new-product requests get three candidates each. The hybrid finds all six and says "none" to all four.
The hybrid wins for two reasons that the output shows separately. Its two signals cover each other's gaps: keyword search finds exact terms and identifiers, such as a request that names REQ-163 directly, and the meaning-based signal finds paraphrases. That matches what larger studies report. The BEIR benchmark found "BM25 is a robust baseline" across 18 retrieval datasets (Thakur et al., 2021), and Anthropic reports that combining contextual embeddings with contextual BM25 "reduced the top-20-chunk retrieval failure rate by 49%" on its own test sets (Anthropic, 2024). But neither signal says "none" on its own. That comes from the threshold.
The test: run your retriever on five requests your analysts say have nothing related. If any comes back with candidates, find out what your agent does with them.
A threshold you can defend
The threshold, 0.67, isn't a number anyone chose. The checker computes it from the calibration split: the midpoint between the highest score any requirement got for a request with nothing related, and the lowest score a truly related requirement got. If those two overlap, so that no threshold separates them, the checker refuses to produce one, and a test proves it does.
Two habits keep the threshold honest. First, it is set on one set of requests and judged on another. Setting and judging a threshold on the same requests flatters it, because it was chosen to fit exactly those. A test in the suite changes every label in the test split and confirms the threshold doesn't move. Second, it is recalculated whenever something it depends on changes: the embedding model, the glossary, or a large change to the repository. New requirements change which ones are nearest to a request, so a threshold that separated the two halves last quarter may not this quarter.
The hybrid still returns 3 extra requirements beside a related one, requirements nobody labelled as related. The checker reports those but doesn't judge them, because a retriever's job is to find candidates, not to decide relations. The agent still has to quote both spans for any relation it proposes, and the blueprint's relation-precision bar, at least 0.90 on the 200-request golden set, is where those extras are judged. The 18 requests here are enough to show the method, not to set a production threshold; Evaluating AI Agents covers sizing a test set for the decision it informs.
The result contract matters as much as the threshold. Every retriever returns {"status": "ok", "results": [...]}, and an empty list means one thing only: the search worked and nothing is related. Stage 5 already made failures into typed errors, so an empty list can't hide a timeout. Requirements below the threshold aren't passed to the agent at all, not even as "weak matches", because a model given a weak candidate tends to use it. The agent's note for an empty result says what was searched and that nothing scored above the threshold, which is something an analyst can check.
With seven new tests, covering both halves of the labelled set, the identifier case, the empty result, the calibration split and the refusal to calibrate, the series suite has 58, and all pass.
The test: find where your retriever's threshold came from. If nobody can say which requests it was set on, or it was set on the same requests it is judged on, it hasn't been calibrated.
Building a retrieval check: a method
Label requests with what is related, and with "nothing". Take closed requests and ask the analysts which records each related to. Done when at least a third of the set is requests with nothing related, drawn from real new work.
Split before you tune. Done when a calibration split and a held-out test split exist, and nothing is tuned while looking at the test split.
Run more than one signal. Keyword search for exact terms and identifiers, embeddings for paraphrases. Done when both are scored on the same requests.
Calibrate the threshold on the calibration split. Done when the threshold is computed, not chosen, and the calculation refuses when no threshold separates the halves.
Judge on the held-out split, and re-run on every change. Done when both bars, found and said-none, run in CI and are re-run whenever the model, the glossary or the repository changes.
Tradeoffs and exceptions
Approach | Gains | Costs | Best suited for | Warning signs |
Keyword search (BM25) | Exact terms and identifiers; explainable | Misses paraphrases; invents on incidental words | Repositories with a controlled vocabulary | Requests in users' words that find nothing |
Vector top-k | Finds paraphrases | Never says "none" | Search boxes where a person reads every result | An agent proposing relations for new products |
Hybrid with a calibrated threshold | Finds both kinds; can say "none" | A labelled set; recalibration | Agents that act on what retrieval returns | A threshold nobody can trace to data |
Hybrid plus a reranker | Better ordering of candidates | Another model to evaluate and run | Large repositories with many near-duplicates | Reranker scores used as a threshold without calibration |
Not every agent needs retrieval. If the repository is small enough to fit in the agent's context, giving it every requirement may be simpler and easier to test than searching; Context Engineering: A Practitioner's Guide covers when that budget runs out. For the mechanics of embeddings and a first retrieval pipeline, see Add Embeddings and Search and Build a Basic RAG System.
How Stage 8 goes wrong
A default threshold. A similarity cut-off copied from a library example or a vendor's documentation is calibrated to someone else's data. Compute yours from your own labelled requests.
"None" softened by the agent. Retrieval returns nothing, and the agent writes "no exact match, but REQ-118 may be relevant" from memory or from an earlier step. The instructions need a case that says what an empty result means, and the Stage 4 spec needs a test for it.
Quiet recalibration drift. The embedding model is upgraded and nobody re-runs the check; scores shift and the threshold no longer separates anything. Pin the model version and re-run on every change, as Stage 7 does for memories.
Grounding mistaken for relevance. A grounding check proves a quoted span exists. It can't prove the span is about the same thing as the request. Relevance is retrieval's job, and the analyst's.
What to do next
Ask your analysts this week for ten closed requests that turned out to have nothing related, and run your retriever on them. If any come back with candidates, add a threshold and calibrate it on half your labelled set. Stage 9 wires the agent's steps into a workflow, from agent to tool to verification to decision to action, where a failed step stops the run or hands it over.
Final takeaway
A retriever that always returns something will always give the agent something to relate. Test retrieval on requests with no answer as well as requests with one, combine exact and meaning-based search, calibrate a threshold on requests you then hold out, and make an empty result mean exactly one thing. Then "no related requirements found" is a finding, not a gap.
Sources
Anthropic, Introducing Contextual Retrieval, 19 September 2024. Failure measured as 1 minus recall@20; contextual embeddings with contextual BM25 "reduced the top-20-chunk retrieval failure rate by 49%" on Anthropic's test sets.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava and Iryna Gurevych, BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models, NeurIPS 2021 Datasets and Benchmarks Track. "BM25 is a robust baseline."
Series navigation. Previous: Stage 7: Decide What It Remembers, and Test What It Forgets. Next: Stage 9: A Workflow Where Every Failure Stops or Hands Over. The full stage list is on the Build an AI Agent From Scratch hub.
Frequently Asked Questions
How do you evaluate retrieval for an AI agent?
Label a set of real requests with the records that are related to each, and include requests where nothing is related. Measure two things on requests held out from tuning: whether each request with a related record finds it, and whether each request with none gets an empty result.
Why does vector search return results when nothing matches?
A top-k vector search returns the k nearest records by similarity, and there is always a nearest record. Without a similarity threshold it cannot return nothing, so an agent will be handed candidates for requests that have no related record at all.
Should an AI agent use keyword search or embeddings?
Often both. Keyword search such as BM25 finds exact terms and identifiers; embeddings find paraphrases. A hybrid of the two with a calibrated threshold can find what is related and still return nothing when nothing is.
Part of the Agent Engineering curriculum: eight paths from prompts and context to testing, evaluation and governance.
