Discovery for AI Features: Scoping Before You Prompt

Scoping an AI feature by describing its output is how you get something faithful, fluent, and quietly unused. The scoping question is which decision the output feeds.

Illustrative composite: Penhale Mutual is ShiftQuality's running case. Its figures are illustrative, not measured.

Penhale Mutual's claims-triage assistant began as one line in a planning document: summarise the FNOL report. Nobody argued. First notice of loss arrives as free text plus a few structured fields, every adjuster reads it, and a summary saves reading.

The first demo produced four-sentence summaries that were faithful and well-formed. Adjusters used them for two weeks, then drifted back to the full report — no complaint, no ticket, just a usage line that sloped down and flattened. Asked why, one was blunt: the summary never told her what she had opened the report to find out.

The unit is the decision, not the output

Discovery for an AI feature has one unit of analysis: the decision the output feeds, and the fact that decision turns on. Everything adjacent belongs elsewhere: what a criterion may assert about one run to acceptance criteria for non-deterministic features, the bar every AI feature clears to the definition of done, which named failures deserve budget to risk-based testing. Discovery asks one thing: when this output appears, who acts differently, and what has to be true for them to act?

"Summarise the FNOL report" fails that with no model involved — it names an artifact, not an actor. The version that names the decision, help an adjuster decide within the hour whether this claim routes to fast-track, standard handling, a documentation chase, or investigation, tells you what the output must contain and what it may drop.

The test: name the person who acts on this output and the window they act in. If you cannot name a human and an hour, you are scoping decoration, not a feature.

When the request arrives as an output ("summarise it", "make it faster"), The Five Whys for Requirements Discovery is a quick route from the artifact back to the decision, provided each answer is checked against the work rather than accepted as the next link.

Trace the decision backwards through closed cases

The move that catches that scope is not a sharper workshop question. It is a backwards trace on finished work: pull the last twenty cases the team closed, ask two things of each — what did you do next, and what made you do it? — then check the part nobody checks. Was it in the input you plan to hand the model?

Penhale ran the trace after the demo went quiet.

What the next action was	Claims	What the decision turned on	In the FNOL report?
Routine — assign and proceed	9	Nothing looked unusual	Yes
Fast-track settlement	3	Value under the adjuster's authority limit	Yes
Request more documentation	2	A missing item (photos, police reference)	Yes
Escalate to special investigations	4	Prior claim history on the same policy	No
Hold — coverage question	2	Policy endorsement added mid-term	No

Six of the twenty turned on something the report does not contain. Claim history lives in the policy record; a mid-term endorsement lives in the policy document. No summary of the report, at any model quality under any prompt, could produce those six decisions — the deciding evidence was never in the room. A model could still say "check the claim history"; it could not say what the history showed, and that is what the adjuster acted on.

That is a ceiling built into the input set, and it is invisible on easy cases: the demos ran on ordinary claims, where any competent summary looks like it is helping. Fourteen of twenty was the honest maximum, and nobody held it because nobody had asked in a form that produces a number. A ceiling is not a reason to stop — it is a reason to change the inputs or the scope.

The test: run the trace on twenty closed cases and count how many turned on a fact absent from the inputs you intend to give the model. That count is your ceiling, before a single prompt exists.

Write the ideal output by hand, on the input you would rather skip

The second artifact is cheaper and less popular: take three to five real inputs, including the one everyone would rather skip, and write the ideal output by hand — the actual words, not a description of them.

Penhale's hard input was a water-damage FNOL where the claimant's account and the contractor's disagreed about when the leak started. It took forty minutes, thirty-five of them argument. Does the assistant flag the contradiction or pick the likelier account? Does the flag go in the routing line or the body? Is it prose, or a field the workflow can branch on?

Three questions that had been vague unease for weeks, settled in an afternoon, because no concrete sentence hedges all three. Hand-writing converts taste into specification, and those outputs become the eval set's first entries.

The test: have two people write the ideal output for the same hard input, separately. If they disagree, you have found the specification, not a style difference. If neither can write one, the scope is too broad — learned for the price of an afternoon.

Name the failures that will still sound right

The characteristic failure of an AI feature is not an error. It is an output that is well-formed, confident and wrong, and it survives review because reviewing it feels like reading rather than testing.

Penhale's list was short and specific: fast-tracks a claim whose value sits just above the authority limit; reports "no prior claims" when the history lookup returned nothing because it errored; renders a contradiction as one confident timeline; recommends investigation on a claim type this team never investigates.

Naming them is half the move — a named failure with no observable is a fear, not a test. The over-limit fast-track is a nightly query. The swallowed lookup error is a null-versus-empty distinction in a log line. The laundered contradiction leaves no signal unless a human reads the source, which makes it a sampling decision, not a monitoring one. Ranking them is separate work; risk-based testing covers that.

The test: for each failure mode, write the signal that would reveal it. Any row where the honest answer is "we would notice" is a failure you cannot detect.

The one-page scope, and what discovery may conclude

All of it fits on one page. If it does not fit, the feature is not scoped; it is still an idea.

AI FEATURE SCOPE: <name>

Decision it feeds:      <the decision this output drives>
Primary user + window:  <who acts, how soon>
Ideal outputs:          <hand-written, real inputs, one hard>
Failure modes:          <each with its observable>
Input-set ceiling:      <n of 20 closed cases turned on an absent fact>
Cost of a wrong output: <who absorbs it>
Human in the loop:      <draft-for-approval | autonomous | advisory>
Out of scope:           <what this refuses to do>
Verdict:                <as scoped | narrower | not buildable>

The last two lines are the ones teams leave off, and they turn the page from a record into a decision. The ceiling converts "we should pull in history at some point" into a number somebody argues with in planning. The verdict makes explicit that discovery may conclude not buildable from these inputs — the most valuable thing it produces, and the outcome most templates leave no room for. The hand-written outputs seed the eval set; how many samples a claim rests on decides how far it must grow.

The test: show the page to someone who was not in the room and ask what the feature refuses to do. If they cannot answer from the page, the "out of scope" line is decorative.

When the backwards trace is the wrong move

Three situations where it earns nothing:

• A genuine capability spike. When the open question is what the model can do at all, you are not scoping yet. Time-box it, expect to bin the code, scope afterwards.

• An internal tool whose only user is its builder. The trace surfaces knowledge in someone else's head. When head and hands are the same, it is theatre.

• A decision already documented and followed. If a written rule governs routing and adjusters demonstrably apply it, the trace only re-reads the rule.

The third is claimed most often and wrong most often.

The test: before skipping the trace, ask someone to produce the document stating the rule, and open it. If nobody finds it in five minutes, or it describes a process that changed two reorganisations ago, the decision is folklore, not policy — and the trace is back on.

What to change this week

Take the AI feature closest to having a prompt and run the trace on twenty closed cases. Ask the two questions, put the answers in a table whose last column reads present in the inputs — yes or no, and count the no's. Then hand-write the ideal output for the hardest of those inputs, with a second person disagreeing with you. You end the day holding a ceiling, a specification and a verdict — none of which prompt iteration produces.

Penhale ended up elsewhere than it started. The assistant reads the policy's claim history alongside the report and leads with a routing recommendation; the summary sits second on the page now, which is where it belongs. The eval set was seeded from the hand-written ideal outputs, which is why it is dense in the eleven non-routine claims — above all the six the report alone could never decide — rather than the nine routine ones every demo used. The original scope was not repaired — it was replaced, by an afternoon of reading closed claims.

Once discovery is done, Writing Requirements for AI/ML Features covers how to write the result down.

Sources

• This post cites no external statistics. Penhale Mutual, the twenty traced claims and the six-of-twenty ceiling are an illustrative composite, built to show the method — not a record of a specific insurer.