top of page

Writing Requirements for AI/ML Features

  • Shawn West
  • Jun 3
  • 11 min read

Updated: Aug 17

A requirement for a deterministic feature names the output. A requirement for a probabilistic one can't — so it has to name something else: the acceptable error profile, and who absorbs each kind of error. Most AI requirements skip that substitution and reach for an accuracy number instead. Here's why that number is almost always the wrong requirement, and the sentence that replaces it.

Every requirement you've written for ordinary software has the same hidden property: it names the output. The system shall calculate sales tax as the taxable amount times the jurisdiction rate. One right answer, described; verification is comparison. That property disappears the moment the system is a model. There is no single right summary of a support ticket, no single right fraud score, no single right ranking. Write the requirement in the usual shape and you get a line that can neither be satisfied nor refuted — so it will never become a decision.

The standard workaround substitutes a number for the output. The model shall be at least 95% accurate. It looks like a requirement: subject, verb, threshold. It is the most common serious defect in AI requirements documents, and the reason is arithmetic.

The requirement a model can satisfy by doing nothing

Accuracy is the fraction of predictions correct across both classes, weighted by how often each class occurs. When one class is rare, that weighting swallows the measurement.

Take card-not-present fraud, where 0.4% of transactions are fraudulent. A model that flags nothing at all — a function returning false unconditionally, no inference, no training run — is correct on 99.6% of transactions. It satisfies "at least 95% accurate" with 4.6 points to spare, and catches no fraud whatsoever. That is the ordinary condition of most business classification problems, where the interesting class is the rare one: fraud, churn, defect, escalation, readmission, safety incident. The rarer the thing you care about, the more completely an accuracy threshold measures its rarity rather than your model.

The base rate is only half the problem. The other half is that accuracy averages two errors that are not alike. A false positive and a false negative land on different people, cost different amounts, and — the part that changes what you build — move in opposite directions when you turn the threshold. Bundle them into one number and you have delegated that tradeoff to whoever tunes the model, usually someone who has never seen the support queue.

Run this today: find the accuracy or "quality" threshold in your AI requirement, look up the base rate of the positive class in your production data, and compute what a do-nothing model would score. If the gap is small, your requirement is measuring prevalence.

One fork, walked

The following is a labeled composite drawn from patterns common in payments and risk work. The volumes and unit costs are illustrative, not measured; the arithmetic is real.

A mid-market payments company — call the program Kestrel — scoped a real-time fraud model for card-not-present transactions. Roughly 2 million transactions a month at a 0.4% fraud rate: about 8,000 fraudulent transactions monthly. The approved requirement read:

REQ-114. The fraud detection service shall identify fraudulent transactions with at least 95% accuracy.

Reviewed by four people. Signed. Nobody objected, because the sentence has the grammar of a requirement.

The model shipped at an operating point flagging about 10,000 transactions a month. Roughly 5,600 were genuinely fraudulent — precision around 56%, recall 70% against the 8,000 real attempts — which means it also declined about 4,400 legitimate transactions. Overall accuracy: 99.66%. REQ-114 was satisfied comfortably, and would equally have been satisfied by the do-nothing model, by one twice as aggressive, and by one half as aggressive. It could not distinguish between any of them.

Those 4,400 declines were not a metric. They were 4,400 people standing at a checkout with a card that just failed. Kestrel's support data put roughly a third of hard declines into a contact within 48 hours — about 1,400 calls a month at a fully loaded handle cost — plus an attrition tail nobody had costed, because no requirement had asked anyone to. The 2,400 missed fraud events, at a few hundred dollars of chargeback exposure each, was a figure Finance already tracked weekly.

One error had an owner and a dashboard. The other had 4,400 victims and no line item. That asymmetry, not the model, is what the requirement should have controlled. The postmortem rewrite:

REQ-114 (v2). At the deployed operating point, the fraud service shall detect at least 75% of fraudulent card-not-present volume by value, while declining no more than 0.25% of legitimate transactions. Both figures are measured monthly on live traffic, reported in aggregate and broken out by issuing country and by account tenure band, with no reported segment exceeding twice the aggregate legitimate-decline rate. Transactions the model flags shall be routed to step-up authentication, not declined outright. Owner of the decline bound: VP Customer Operations. Owner of the detection bound: Head of Risk. Review date: 90 days after launch, then quarterly.

Read what that sentence forced into existence. The 0.25% ceiling was tighter than the model could hit at any threshold that also met the detection bound — exactly the information a requirement is supposed to surface. The resolution was not a better model. It was a step-up authentication flow that had appeared in no scope document, because as long as the requirement said "95% accurate," nobody had to confront the fact that a probabilistic output was wired straight to an irreversible customer-facing action. The requirement changed the architecture, which is the test of whether a requirement is real.

Run this today: write down the raw monthly count of each error type at your model's current operating point, not the rate. Rates are arguable; "4,400 people" ends the argument.

The discovery move: who receives the mistake?

The question that would have caught REQ-114 is not a data-science question and cannot be answered by the model team:

"For each kind of mistake this model can make, who receives it, what does it cost them, and can they appeal it?"

Who receives it forces a person rather than a metric. What does it cost them makes the two error types comparable in the only unit that matters. Can they appeal it separates a recoverable inconvenience from a terminal one — the largest single driver of how tight the bound must be.

At Kestrel that question had an available answer the whole time: support knew what a decline callback cost, Risk knew the average chargeback. Nobody put the two numbers in the same room, because intake treated "fraud detection" as one requirement instead of two error budgets with two owners — a discovery gap of exactly the kind scoping an AI feature before you prompt is meant to close. Written out, the answers become the spine of the requirement:

Error type

Who absorbs it

What it costs them

The clause that bounds it

False positive — legitimate transaction flagged

The cardholder, at checkout

Failed purchase, a callback, possible churn; unrecoverable in the moment

≤0.25% of legitimate transactions declined, monthly, on live traffic

False positive, concentrated in one segment

Cardholders in a country or tenure band

The same cost, but systematically, to an identifiable group

No reported segment above 2× the aggregate decline rate

False negative — fraud not flagged

The business, then the issuer

Chargeback exposure plus dispute handling

≥75% of fraudulent volume by value detected

Abstention — score in the uncertain band

The review analyst

Queue time; a call made under time pressure

Uncertain band routed to step-up auth; queue ≤200 items/day

Silent decay — the profile drifts post-launch

Everyone, invisibly

Bounds stop holding while the dashboard stays green

Monitored floor with alerting; 90-day review, then quarterly

The table does what an accuracy threshold cannot: it makes the tradeoff negotiable by the people who pay for it. Customer Operations can argue for 0.15%; Risk can argue 75% is too loose. That argument is the requirement being written. When the document says "95% accurate," the argument never happens and the threshold gets set by default.

Run this today: draw this table for your feature with every column blank except "who absorbs it." If any row's owner is "the model team," that error has no real owner.

The five clauses

Generalizing from REQ-114 v2, a usable probabilistic requirement carries five parts. Drop one and it degrades back toward an accuracy number.

  1. The decision the output informs. Not "the model shall predict churn risk" — the score shall determine which accounts enter the retention outreach queue. A model informing no decision needs no requirement, because nothing can be wrong.

  2. The error type and its bound, at a stated operating point. Precision, recall, false-positive rate — named individually, with numbers, tied to the deployed threshold. A metric quoted without an operating point is a curve, not a commitment.

  3. The population it holds over. Aggregates average across groups, and averages conceal. Buolamwini and Gebru's Gender Shades audit is the clearest demonstration: commercial gender-classification systems with respectable headline accuracy showed error rates up to 34.7% for darker-skinned women against 0.8% for lighter-skinned men. Both sit inside the same "accurate" model. Mitchell et al.'s Model Cards work turns that into a reporting discipline — evaluation disaggregated by group, not only in aggregate.

  4. The fallback path. What happens when the model abstains, errs, or is unavailable — where the requirement stops being about the model and becomes about the system around it.

  5. The monitored floor and the review date. Code does not decay; models do, because the world their inputs describe keeps moving.

Clauses 2 and 3 are where teams reach for acceptance criteria for non-deterministic features, which covers turning these bounds into criteria you can verify. The requirement names the bound; the acceptance criteria establish how you'd know it held.

Run this today: check your AI requirement against the five clauses. In our experience clause 3 is missing outright and clause 1 appears only as a noun ("churn scoring") rather than a decision.

The threshold is a business dial, not a model property

A trained model does not have a precision and a recall; it produces scores. The threshold — the cut point where a score becomes an action — selects which pair of error rates you get, and that pair moves along a curve. Raise it: fewer false positives, more missed fraud. Lower it: the reverse. The model team owns the curve; the business owns the point on it, because the point is a statement about whose mistake is cheaper. "95% accurate" hands that choice, silently, to whoever runs the training script.

Calibration matters when the score is believed numerically rather than used as a cut. A model can rank well and still be badly calibrated — its 0.9 scores right only 60% of the time. Ranking suffices if you take the top N; it doesn't if the score feeds an expected-cost calculation, a quote, or a triage priority. Treat output as a probability and the requirement needs a calibration clause.

Run this today: ask who chose the current threshold and what they were optimizing. "It was the default" or "it maximized F1" means the operating point was set by a library, not by your business.

The clause everyone forgets: decay

Deterministic code fails the day you break it. Models fail slowly, as inputs drift from the training distribution — new fraud tactics, a new customer mix, an upstream field that quietly changed units. Sculley and colleagues' Hidden Technical Debt in Machine Learning Systems named the structural version: in an ML system, changing anything changes everything, because features are entangled and no input is genuinely independent. The "small" retrain, the new data source, the upgraded upstream model can each move your error profile without a code review noticing.

So the requirement needs its own expiry: a monitored floor (an alert threshold on the bound, with an owner and a stated action) and a review date (a calendar commitment to re-measure against fresh labels, not a promise to monitor).

Regulation is converging on the same shape. The EU AI Act requires high-risk systems to achieve appropriate accuracy, robustness and cybersecurity and to perform consistently throughout their lifecycle, with declared accuracy levels and relevant metrics stated in the instructions for use (Article 15). NIST's AI Risk Management Framework puts the same discipline in its MEASURE and MANAGE functions. Neither tells you what your number should be; both say a number without a lifecycle isn't a commitment.

Run this today: for your deployed model, find the date the current error rates were last measured against a labeled sample. If it's the launch date, you don't have a monitored requirement — you have a launch memory.

The exception: when the deterministic requirement is still right

None of this argues for wrapping every requirement in probability language. Two exceptions matter, and the second is the larger one.

Keep the deterministic requirement when the rule is knowable. If the decision can be written as a policy — eligibility thresholds, tax rules, entitlement logic, safety interlocks — write the policy. A rule is auditable, explainable to a regulator, unit-testable, correctable in an afternoon, and does not drift. Teams routinely reach for a model on problems whose real requirement is fifteen lines of business logic nobody wanted to elicit: a discovery failure dressed up as an ML project. The line between the policy the business owns and the behavior the system implements is the ordinary business, user, and system requirement split, and a model in the middle doesn't erase it.

Don't ship a model at all when these hold together:

  • The error is unrecoverable for whoever receives it. A wrongly denied claim, a wrongly flagged benefits application, a wrongly rejected candidate. Without a cheap appeal path, the false-positive bound you'd need sits below what the model delivers at any useful recall.

  • You cannot state the error profile. No labels, no ground truth, no way to get either means you cannot write clause 2, so you cannot write the requirement. Ship the human process and start collecting labels.

  • The costs are asymmetric enough to have one right answer. If a false negative costs a thousand times a false positive, you don't need a model to locate the operating point. You need a conservative rule and a review queue.

A milder case: when the error rate is uninteresting because a human edits every output before it acts. A drafting assistant reviewed before sending has a soft usefulness bound rather than a correctness bound — the shape covered in functional vs non-functional requirements.

The diagnostic

Take the AI requirement you're closest to shipping and spend twenty minutes:

  1. Compute the do-nothing baseline. With the base rate in hand, what does a model that always predicts the majority class score against your threshold? Any requirement that baseline satisfies is not a requirement.

  2. Name both error types in raw monthly counts, and next to each, the person who receives it. Blank cells are unasked elicitation questions.

  3. State the operating point. Which threshold is deployed, who chose it, what were they optimizing? A library default means the tradeoff hasn't been made yet.

  4. Break the aggregate. Split your headline metric by your two most operationally meaningful segments. Any segment above twice the aggregate error rate is a requirement clause you can write today.

  5. Put a date on it. Add the review date and the alert floor to the requirement text itself, with an owner's name attached.

"The model shall be 95% accurate" survives review because it looks like every requirement anyone has ever approved. What it does is hide a business decision — which mistake we'd rather make, and to whom — inside a technical-sounding number, then hand that decision to whoever tunes the threshold. The rewrite is longer, uglier, and carries two owners' names. It is also the only version that could tell Kestrel, before a line was built, that what the requirement really needed was a step-up authentication flow.

Sources

bottom of page