Responsible AI Beyond the Checkbox

Responsible AI is usually organised around the user. For most consequential systems the user is not the person the decision happens to, and that single mismatch is what makes the mechanisms decorative.

Penhale Mutual's fast-track model decides which claims settle in days and which enter standard handling and settle in weeks. A handler sees the recommendation. A claimant sees a settlement that is either quick or not.

The responsible-AI work had been done conscientiously. Principles published, an ethics review at design, a model card, a feedback mechanism.

The feedback mechanism was a flag button in the handler's console, routing to the model team's backlog. Over eight months it was pressed four times.

That is not apathy. Handlers had no reason to flag: the recommendation looked reasonable on the claim in front of them, and the cost of a wrong one landed on somebody who was not in the room and had no idea a model was involved. Every mechanism the team built was pointed at the user. The harm was pointed at the claimant.

The unit is the person the output lands on

Start by separating two roles that most designs collapse. The user operates the system. The subject is whoever the output is about, and bears the consequence.

For a coding assistant these are the same person, which is why the intuitions built on those tools transfer so badly. For a claims model, a hiring screen, a credit decision or a content moderation system they are different people — and only one of them is in your product analytics.

	Uses it	Bears the outcome	Knows a model was involved	Can contest it
Handler	Yes	No	Yes	Yes, trivially
Model team	No	No	Yes	Yes
Claimant	No	Yes	No	No route exists
Regulator	No	No	On request	After the fact

Read the bottom-left cell against the bottom-right. Everything the team built lives in a surface the claimant never touches.

Some harms never have to reach the subject at all — that is what the runtime interception layer is for, and it is the cheapest place to stop the ones it can recognise. This post is about the rest, which is most of them.

The test you can run: for your system, name the person who bears the cost of a wrong output. Then find the mechanism they use to say so. If it's the same mechanism your users have, check they can actually reach it.

The review happens after the decisions that determine the outcome

The second structural problem is timing. An ethics review at the end of design arrives after the training data has been chosen, the objective has been defined, and the population the system will run on has been fixed.

Those three choices contain most of what a review is looking for. Asking whether the finished model is fair is a question about decisions made months earlier by people who weren't framing them as ethical ones — they were framing them as scoping, which is exactly what they were.

That's the useful reframe: this is the same work as discovery, asked with a different question. Discovery asks what decision the output feeds. Add one more: who is on the other side of that decision, and what does a wrong one cost them? Asked at that point it's cheap and it changes the design. Asked at review it changes a document.

The test: find the date of your ethics review and the date the training data was selected. The gap between them is the portion of the design the review could not influence.

Aggregate accuracy hides disparate impact by construction

This is arithmetic rather than ideology, and it's worth doing once so the shape is obvious.

Suppose a subgroup is 6% of your population and the model is 20 points less accurate for them. The aggregate moves by 0.06 × 20 = 1.2 points — well inside the range a team would read as noise between model versions. The group's experience is dramatically worse and the headline number barely twitches. The smaller the group, the more completely the average conceals them, which inverts the intuition that small groups are low-stakes.

Penhale's fast-track model was accurate overall. Disaggregated by intake channel, phone-submitted claims were referred to fast-track far less often than online ones — because phone intake produced shorter free-text, and the model had learned to read length as complexity. Nothing about the model was aimed at older claimants. The effect was that claimants who phone rather than submit online waited longer to be paid.

The honest complication, which the principles page never mentions: measuring this requires knowing something about your subjects, and collecting that data creates its own risk. There's no clean resolution. What's not defensible is using the tension as a reason not to measure — start with attributes you already hold for operational reasons, like channel, region or product, which are often strong enough proxies to surface the shape.

The test: re-run your headline metric split by three attributes you already hold. If any split shows a gap you'd consider unacceptable as a headline number, you've been shipping it inside an average.

The contest path is the part nobody funds

Building the model is a project with a budget, a team and a date. Building the route by which a person affected by it can say this is wrong is an afterthought — unowned, unfunded, and quietly dropped when the launch date tightens.

A functioning path has four links, and it fails at whichever is missing:

• A surface the subject can reach. Not the operator's console. For Penhale that meant a line on the settlement letter saying how the timeline was set and how to ask for it to be looked at again.

• A human with authority and context. Someone who can actually change the outcome for that person, not log it.

• A route back to the model team. Individual corrections that never aggregate teach you nothing.

• A threshold that triggers action. A pattern in contests has to be able to change or stop the model, or the loop terminates in a spreadsheet.

The four presses in eight months were not a signal that the model was right. They were a measurement of the mechanism, and it read zero.

The test: trace one real complaint end to end. Where it stops is where your loop actually ends, regardless of the diagram.

When the checklist is the right instrument

Not all of this is warranted everywhere, and applying it uniformly is how it becomes the thing people route around:

• Subject and user are the same person. A developer tool where the only person affected is the one typing needs far less of this — they have a contest path called not using it.

• The output is trivially reversible and immediately visible. A wrong autocomplete is corrected in a second by the person who saw it.

• A regulated floor already applies. Where a standard specifies required assessments, that's the floor. Do it, then do the work that actually changes the design.

• The system genuinely decides nothing. Ranking search results within a session you can re-run is not the same as deciding a settlement timeline.

And a checklist has one honest use even where it's insufficient: it makes an omission visible. A box left unticked with a name against it is a decision. That's a real improvement on nothing — it's just not the same as the work.

The test: ask whether anything in your responsible-AI process has ever changed a design. If it has only ever produced documents, it's running too late to do anything else.

What to change this week

Don't convene a review. Find the subject.

Write down who bears the cost of a wrong output, whether they know a model was involved, and the exact route they'd use to challenge it. That's a twenty-minute exercise and for most systems it produces a blank cell — which is the finding, and it's a product gap rather than an ethics one.

Then split your headline metric by one attribute you already hold. Channel, region, product, tenure. You're not looking for a verdict, you're looking for whether a gap exists that the average has been absorbing.

Where those two turn up something real, it needs a record and a threshold, not a principle. Penhale's settlement letters now say how the timeline was set. Contest volume went from four in eight months to a few dozen a month, which looked like a regression on somebody's dashboard for exactly one meeting.