The Beacon Model: Why AI Should Amplify, Not Replace

Amplify-not-replace is usually argued as a values position. It is more useful as a design variable — one you set per decision, and one most teams set by accident.

Penhale Mutual's fraud model scores every claim for referral. The design document is unambiguous: the score is advisory. A referral analyst reviews the claim and decides.

Someone finally measured it. Across a quarter, analysts agreed with the score 99.2% of the time.

There are two readings. Either the model is extraordinary and the analysts are a formality, or the analysts are not really deciding. The team assumed the first and checked anyway, because 99.2% is a suspicious number — it's the agreement rate you'd expect from a system where the human's job has quietly become confirmation.

The cause was in the interface. The score appeared at the top of the claim file, before the analyst had read anything. By the time they formed a view, they had one.

The unit is one decision, and its autonomy level

A lighthouse doesn't sail your ship; it shows you where the rocks are. That's a good image and it isn't a design. The design question is narrower and answerable: for this specific decision, what is the system allowed to settle, and what must a person settle?

That's a variable with a small number of settings, and it's set separately for each decision a system touches. Penhale's assistant summarises (autonomy: none), scores fraud risk (claimed: advisory), and recommends a handling path (autonomy: nobody had ever asked). Treating "is this system replacing people" as one question about the product is how two of those three end up unset.

The test you can run: list the decisions your system touches and write the intended autonomy level beside each. The ones you can't answer are set by whatever the code happens to do.

"Human in the loop" is a claim, and it has a number

The loop is only a control if the human sometimes changes the answer. That makes it measurable, which is unusual for anything in this territory.

Level	What the person does	The number that tells you it's real	Penhale
Advisory	Forms their own view, then sees the model	Disagreement rate — and whether disagreements get investigated	Claimed here
Recommendation	Sees the model, then confirms or overrides	Override rate. Near-zero means it isn't this level	Actually here, at 0.8%
Default	Model acts unless a person intervenes	Intervention rate, and time-to-intervene	Handling path, unnoticed
Autonomous	Nothing, by design	Post-hoc audit rate	Summarisation, correctly

The second row is where most systems live while their documentation claims the first. And the distinguishing question is not how the team describes it — it's what the person sees first.

A number worth holding on to: an override rate near zero is not evidence of a good model. It is evidence that you cannot tell, because a rubber stamp and a genuinely excellent system produce identical logs. (The release-gate version of this is the mirror image — there, an override rate near one is the tell.)

The test: compute your human-override rate over a quarter. Under about 2% on a judgment task, assume you have a recommendation system with a signature step until you can show otherwise.

What a reviewer needs in order to disagree

Calling something advisory doesn't make a person able to dissent. Four things have to be true, and interfaces routinely break the first without anyone deciding to:

• They form their own view first. A model output shown before the human's judgment is an anchor, not advice. Penhale moved the score behind a click, after the analyst records their own call. Agreement fell from 99.2% to about 87% — and the referrals that came out of that 13% were the point of having analysts at all.

• They have the information the model used. "High risk, 0.86" is unarguable. Prior claim on the same policy, contractor named on two other open claims can be checked, and therefore contradicted.

• They have the time. A queue sized on the assumption that most cases get confirmed makes disagreement the expensive option. Throughput targets set the autonomy level more often than design documents do.

• Disagreeing is not penalised as error. If overrides are reported as deviations, the rate goes to zero within a quarter and stays there.

The first is the cheapest to fix and the one most often wrong, because showing the answer immediately is the obvious interface.

The test: watch someone use it. If the model's answer is on screen before they have formed a view, you built a recommendation system whatever the document says.

Amplification costs more, and that is the real argument

The honest case for keeping people in the loop is not that it always produces better outcomes. It's that it produces different errors, and errors a person can be accountable for.

Whether the pairing beats either alone is frequently asserted and rarely checked locally. It's also directly measurable in your own system: sample cases where the human and the model disagreed and have a senior reviewer adjudicate. That tells you which of the two was right more often in your setting, which is the only version of the claim that can inform a decision.

Do that and the cost becomes explicit. Analyst time is real, the queue is real, and the disagreement rate you're paying for has a value you can now name. That's a better conversation than a principle — it's the one where somebody can decide to buy less of it deliberately.

The test: take fifty cases where a person overrode the model and have someone senior score who was right. If the human wins meaningfully more often than chance, you have the value of the loop in a number. If they don't, you have a finding worth acting on.

When replacement is the right call

Amplification is a default, not a rule, and applying it everywhere is how it gets a reputation as overhead:

• No judgment is involved. Sorting, formatting, deduplicating, transcribing. Inserting a person adds latency and an opportunity for error.

• The volume makes review fictional. A person nominally reviewing 4,000 items a day is a compliance artifact. Either sample honestly or automate honestly, but don't pretend.

• The person has no basis to disagree. If the model uses signals a human can't perceive, "review" is a coin flip wearing a lanyard. Fix the explanation or accept the autonomy.

• The reversal cost is trivial. Where a wrong output is noticed immediately and undone in seconds, the loop can move after the fact.

• The human is the bottleneck on something that matters more. Sometimes the responsible move is to automate the routine tier so people can spend attention on the tier that needs it.

The test: for each decision you keep a human on, name what they contribute that the model can't. If the answer is "accountability," that's legitimate — but say it plainly, because it means the loop is there for who answers, not for accuracy.

What to change this week

Don't debate the philosophy. Measure one rate.

Take a system you describe as having a human in the loop and compute how often the human changes the answer. It's usually one query. If it comes back under a couple of per cent, look at the screen next — the odds are the model's answer is displayed before the person has formed one, and that single ordering decision is doing more to set your autonomy level than any document.

Then write the intended level next to each decision the system touches, and give the ones nobody has set to whoever owns the harm those decisions cause. Where the level is genuinely high, that's a decision that needs a record and a gate, not a paragraph of intent.

Penhale's fraud score now appears after the analyst logs their own view. Agreement runs in the high eighties, the disagreements are sampled weekly, and the model has been retrained twice on cases the analysts got right. The lighthouse doesn't sail the ship — but it only counts as a lighthouse if the captain can still see the water.