top of page

Discovery for AI Features: Scoping Before You Prompt

  • Shawn West
  • 4 days ago
  • 5 min read

The feature was going to "summarize support tickets." Everyone nodded — it's obvious what that means, right? Two weeks later the demo produced summaries that were technically accurate and completely useless: they compressed a furious three-email escalation into "Customer has a question about billing." The engineer had built exactly what was asked. Nobody had said the summary's whole job was to tell a support lead how angry the customer is and whether it's about to churn — because nobody had that argument before the prompting started. So the model optimized for the one thing it could infer from a vague ask: brevity.

That is the signature failure of AI work. It doesn't fail in the model. It fails in discovery — the same place non-AI projects fail — except AI hides it longer, because a plausible-looking output feels like progress even when it's answering the wrong question. This guide is a discovery method for AI features: what to pin down before anyone writes a prompt, so "good" is a decision you made on purpose instead of one the model made for you.

Why AI makes bad discovery more expensive, not less

With a normal feature, vague scope surfaces fast — the thing either does the workflow or it doesn't. With an AI feature, vague scope produces something that looks done. The demo runs. The output is fluent. Stakeholders see fluency and assume correctness, and the gap between "sounds right" and "is right" only shows up in production, on the inputs nobody tried.

So the discipline that used to be optional-but-nice for a CRUD feature becomes load-bearing for an AI one. You are not scoping a behavior you can eyeball. You are scoping a judgment — and you have to write the judgment down before you can tell whether the model is making it well.

The five questions to answer before you prompt

Run these in a discovery session with product, engineering, and whoever owns the outcome. If you can't answer one, that's not a delay — that's the session doing its job.

1. What decision or action does this output feed? Not "summarize tickets" — "help a support lead decide which tickets to escalate in the next hour." An AI output that isn't wired to a decision is decoration. Naming the decision tells you what the output must contain (in the example: sentiment, churn risk, and the specific ask) and what it can safely drop.

2. What does a good output look like — on a real, hard input? Pull three to five actual inputs, including a nasty one, and write the ideal output by hand. This is the single highest-leverage thing you can do. You now have a target, a shared definition of "good," and the seed of an eval set — before a line of prompt exists.

3. What does a bad-but-plausible output look like? This is the AI-specific question, and the one teams skip. Fluent-and-wrong is the default failure mode. Write down the failures that will still sound confident: the summary that misses the anger, the answer that invents a policy, the extraction that quietly drops a row. If you can't name them now, you won't catch them in review.

4. What's the cost of being wrong, and who absorbs it? A wrong movie recommendation costs a shrug. A wrong dosage summary costs a person. The cost of error sets everything downstream — how much human review you need, whether you ship at all, what your quality gate has to prove. Scope this now or discover it in an incident.

5. What's the human's role in the loop? Is the AI drafting for a human to approve, or acting on its own? "Draft for approval" and "autonomous action" are different products with different risk profiles and different definitions of done. Decide it here, not by accident later.

The one-page AI feature scope

Capture the answers on a single page. If it doesn't fit on a page, the feature isn't scoped yet — it's still an idea.

## AI Feature Scope: <name>

Decision it feeds:   <the human decision or system action this output drives>
Primary user:        <who acts on it>

Good output (examples):
  - Input:  <real hard input #1>
    Ideal:  <hand-written ideal output>
  - Input:  <real hard input #2>
    Ideal:  <hand-written ideal output>

Failure modes to guard (plausible-but-wrong):
  - <e.g. omits sentiment / churn signal>
  - <e.g. invents a policy that doesn't exist>
  - <e.g. silently drops list items>

Cost of a wrong output:  <low / medium / high — and who absorbs it>
Human in the loop:       <draft-for-approval | autonomous | advisory>

Out of scope (for now):  <what we are deliberately NOT doing>
Success signal:          <how we'll know in production it's working>

Notice what this page is not: it isn't a prompt, a model choice, or an architecture. Those are downstream. Scope is about the judgment, not the implementation. Decide the judgment first and the implementation questions get easier; decide the implementation first and you'll retrofit the judgment to whatever the model happened to do.

The discovery-to-eval handoff

Here's the payoff that non-AI discovery doesn't get. The hand-written "good output" examples from question 2, and the "bad-but-plausible" list from question 3, are not throwaway artifacts. They are your first eval set. The moment you have them, you can measure whether a prompt is getting better or just getting different. Teams that skip discovery don't just ship vaguer features — they ship features they can't measure, because they never wrote down what they were measuring against.

Discovery for AI isn't paperwork you do before the real work. It's the thing that makes the real work testable.

Common objections (and why they don't hold)

"We'll figure out 'good' by iterating on the prompt." You'll iterate toward something, but without a target you're optimizing vibes. Iteration needs a direction, and the direction is the thing you refused to define.

"The inputs are too varied to write examples." Then that's the finding: the scope is too broad to build well, and you've learned it for the cost of an afternoon instead of a quarter. Narrow it.

"This slows us down." It moves the slow part earlier, where it's cheap. The alternative isn't fast — it's a demo that looks done, a month of "why is it doing that," and a rewrite once someone finally writes down what "good" was supposed to mean.

Key Takeaway

AI features fail in discovery, not in the model — and AI hides the failure longer because fluent output feels like progress. Before anyone writes a prompt, answer five questions: what decision the output feeds, what good looks like on a real hard input, what bad-but-plausible looks like, the cost of being wrong, and the human's role in the loop. Capture it on one page, and treat your hand-written "good" and "bad" examples as the first eval set. The support-summary demo didn't fail because the model was weak. It failed because "summarize tickets" was the whole spec, and brevity was the only thing left to optimize once nobody said what the summary was for. Scope the judgment first. The prompt is the easy part.

bottom of page