top of page

Data Governance for AI Delivery: Data Is the First Quality Gate

Shift Quality Editorial
59 minutes ago
13 min read

A model evaluation tells you how the system behaves on the data you think you have. Production runs on the data you actually have. Data governance for AI is the work of closing the distance between the two, and it starts before anyone writes code.

The support assistant passed everything. A mid-sized B2B software company had built a retrieval-augmented assistant over its help centre and internal knowledge base, to answer customer questions in the support portal. (Composite scenario, drawn from patterns that recur across organisations; not a single verified incident. Figures are illustrative.)

The team did the work most guidance asks for: a golden set of 300 real customer questions, scored for groundedness, accuracy and refusals, plus a prompt-injection red team. It cleared every threshold, and the launch review signed it off.

In its third week, the assistant told an enterprise customer they were entitled to a service credit under terms withdrawn eighteen months earlier. The account manager found out when the customer quoted it back in a renewal call. Over the next ten days the team found four more things:

  1. The knowledge-base export included an Archive folder of superseded policy pages. Nobody owned the archive, so nobody had been asked whether it should be included.

  2. The help-centre platform had renamed the field that marked articles as draft or retired. The ingestion filter matched on the old field name, found nothing to exclude, and let everything through. Nothing errored.

  3. The nightly index rebuild had grown from roughly 4,000 documents to 6,500 in one night. No check looked at the count.

  4. Some internal troubleshooting notes in the same export named customers and their incidents. Those were now retrievable by any customer who asked the right question.

And when legal asked which document produced the wrong answer, nobody could say which version of which page had been in the index that day. It took a week to reconstruct.

None of that is a model failure. The model did what retrieval-augmented models do: it answered from what it was given. The evals were real and they passed honestly. They were run against a curated set of current articles, which is the data the team believed the assistant would use. The failure was in the data under the system, and every part of it could have been caught by a question asked at intake.

Why the model-first instinct is attractive

When a team is told to make an AI feature safe, it reaches for the model. That is reasonable. Model behaviour is the novel part, eval tooling is mature, and an eval score is something a launch review can read. Data feels like somebody else's problem.

There is a quieter reason too. In conventional software, data problems usually announce themselves: a renamed column breaks a query and a test goes red. In a retrieval-based AI feature, the same change often produces no error at all. The system keeps answering, fluently, from whatever it now has.

To test your own launch process, take the last AI feature you approved and count the review items about the model and those about its data. If the second number is zero or one, your gate checks the engine and assumes the fuel.

What the common explanation misses: evals are conditional on the data

An evaluation result is a statement of the form "on this data, the system behaves like this." Change the data and the statement no longer applies. That is obvious when written down, and easy to forget in practice, because the eval data and the production data usually come from the same place at first and then drift apart.

NIST's AI Risk Management Framework names this directly as one of the ways AI risk differs from conventional software risk: datasets "may become detached from their original and intended context or may become stale or outdated relative to deployment context", and "harmful bias and other data quality issues can affect AI system trustworthiness" (NIST AI 100-1, Appendix B, 2023).

The mechanism has three parts:

  • Data changes on its own clock. Code changes go through a release. A source system's content and schema change whenever its owners change them, with no release in your pipeline. The support assistant's code did not change between eval and incident. Its data changed every night.

  • AI systems absorb bad data quietly. A retrieval index embeds a superseded policy or a confidential note as readily as a good one. Nothing rejects the input unless you build something that does.

  • Accountability for the data sits outside the AI team. The people who could say "the Archive folder is obsolete" or "those notes are confidential" are rarely in the delivery team's meetings.

That third point is the discovery failure underneath the other two. Nobody asked the help-centre owner which content is authoritative, how retired content is marked, and what in the export is not for customers. Each answer would have changed the build.

The discovery move: treat each data source as a dependency with an owner, a promise and a risk, and settle those before design, the same way you would settle an external API before integrating with it.

The six controls, mapped to the delivery lifecycle

Data governance can sound like an enterprise programme with a council and a catalogue. For an AI feature it is narrower and more practical: six controls, each placed at the point in delivery where it is cheapest to apply and each answering one question.

#

Control

Lifecycle stage where it sits

The question it answers

What it produces

1

Intake

Discovery, before the project is approved

Is this data fit, available and permitted for this purpose?

An intake record per source, and a go/no-go

2

Ownership

Discovery, then for the life of the feature

Who decides what this data means, who can change it, and who answers when it is wrong?

A named owner and steward per source

3

Contracts

Design, before integration is built

What does the producer promise, and how will we know when that promise breaks?

A written contract with checks you can run

4

Quality gates

Build and every pipeline run (ingest, training, index rebuild)

Is this batch of data good enough to use right now?

Automated checks that block or warn

5

Lineage

Every pipeline run, used in operations

Which source, version and owner produced this output?

A record linking outputs to inputs

6

Classification and retention

Before ingest, enforced in operation

Who may see this, how long may we keep it, and how do we delete it everywhere?

A classification, access rule and retention period per source

The order matters. Controls 1, 2 and 6 are decisions, and they are cheap at discovery and expensive after launch. Controls 3, 4 and 5 are mechanisms, and they are what turn those decisions into something enforced on every run rather than remembered at kickoff.

The six data governance controls placed along the AI delivery lifecycle, from intake and ownership at discovery, contracts at design, quality gates and lineage at build and every pipeline run, to classification and retention set before ingest and enforced in operation.

The support assistant, replayed with the controls in place

Walk the same project again, control by control, to see where each failure would have been stopped and what it would have cost. (Composite, continuing the scenario above.)

Intake. At discovery, the delivery lead fills in one intake record for the knowledge-base source. The record asks: what is the authoritative set, how is retired content marked, and is anything in this export not for customers? The help-centre owner answers in a 30-minute call. The Archive folder is out. Retired articles carry a status field. The internal notes space is staff-only. Three answers, one call, and four of the five incidents never get built in.

Ownership. The Archive folder had no owner, which is why no one excluded it. The intake record needs a named person, not "the support team", and unowned data stays out of the index until someone accepts it.

Contracts. This is the fork that actually broke production: the renamed status field. With a contract, the dependency is written down. Here is an illustrative fragment. The field names are ours; the Open Data Contract Standard defines a fuller structure for the same ideas, including schema, data quality, SLA and roles sections.

source: help-centre-articles
owner: Head of Customer Education        # named person in the real record
consumer: support-assistant-index
schema:
  - field: article_status
    type: enum
    allowed: [published, draft, retired]
    required: true
semantics:
  - only article_status == published is customer-facing
freshness: export completes by 02:00 UTC daily
change_notice: 10 working days for any schema change, to the consumer channel
checks:
  - article_status present on 100% of rows
  - published share between 55% and 75% of rows

The contract does two jobs. Socially, it tells the producer that an AI feature depends on that field, so a rename becomes a conversation instead of a surprise. Mechanically, the first check fails the moment article_status disappears. The filter's silent no-op becomes a red build.

Quality gates. The contract checks run as a gate before the nightly index swap. One more check catches the growth spike: if the document count moves by more than an agreed margin from the last good build, keep yesterday's index live and alert the owner. That is fail-closed, and its cost is staleness on a bad night. For a support assistant that is the right trade. Where freshness is the point, such as stock levels, warn-and-flag may be better, and that choice belongs in the contract.

Lineage. Each index build records which documents went in, at which version, from which owner, and each answer logs the chunk identifiers it retrieved. Support can then go from the customer's conversation to the chunk, the document version and the owner in minutes rather than a week. The OpenLineage specification offers a shared vocabulary (jobs, runs, datasets and metadata facets), but a small team can start with a build manifest and answer logs.

Classification and retention. The internal notes space is classified before ingest as staff-only, containing customer personal data. It does not go into a customer-facing index at all. If a staff assistant is built on it later, its access rules must carry into the index, and deletion must reach the index and logs as well as the source.

Replayed this way, the project costs perhaps a week of extra discovery and build effort, mostly one-off. The incident cost ten days of firefighting, a disputed renewal and a personal data exposure for legal to assess. Each control is cheapest at the stage the table puts it.

How to evaluate your own position: trace one answer

You don't need a maturity model to find out where you stand. Take one AI feature in production. Pick one real output from last week: an answer, a classification or a recommendation. Then try to answer six questions, one per control, from records rather than by asking around:

  1. Intake: Is there a written record of why this source was judged fit and permitted for this use?

  2. Ownership: Can you name the person, not the team, who owns the source this output drew on?

  3. Contracts: Is there a written promise from the producer about schema, meaning and freshness, and a channel where changes are announced?

  4. Quality gates: Did an automated check run on the batch that produced this output, and would it have blocked a bad batch?

  5. Lineage: Can you name the exact source documents or records, and their versions, behind this output?

  6. Classification and retention: Do you know the classification of that source, who was allowed to see it, and how long it is kept, including in the index and in logs?

Recommended threshold (ShiftQuality's judgement, not an external standard): one point per question answered from a record within 30 minutes. Five or six means the controls exist and are used. Two or fewer means the feature's assurance rests on model evals alone, which, as the support assistant showed, may describe a system that no longer exists.

Minimum viable data governance, for a team without a data office

Most mid-market organisations have no data office or catalogue, and an AI feature cannot wait for one. Here is the smallest credible version.

Control

Minimum viable (one team, one feature)

Fuller version (data office, many features)

Intake

One-page intake record per source, completed at discovery

Intake integrated into portfolio approval

Ownership

A named owner per source, recorded on the intake record

Owner and steward roles across a data domain model

Contracts

A short written contract, checks in the pipeline's CI

Machine-readable contracts in a registry, versioned

Quality gates

Two or three fail-closed checks on the critical fields and counts

A data observability platform with profiled baselines

Lineage

Build manifest per run plus retrieved-chunk IDs in answer logs

Automated lineage across pipelines, e.g. via OpenLineage

Classification and retention

Classification and retention period per source, set before ingest

Enterprise classification scheme, automated retention

The minimum viable column is achievable by one delivery team in a few weeks, because it is mostly decisions written down plus a handful of pipeline checks. The AI Data Intake & Contract template covers the first three rows in one document, and the AI Data Handling Policy gives the classification and retention row a policy to point at.

The trade-off is consistency. Feature-team governance is fast but drifts: the warning sign is two features ingesting the same source under different rules. That is the moment to agree a shared standard, with teams still doing the work, rather than build a central queue that teams will route around.

Exceptions worth making explicitly:

  • Prototypes on synthetic or public data need intake and classification only. Don't build contracts for a two-week experiment, but write down that it is one, so it can't drift into production without the rest.

  • Vendor-hosted features you cannot see inside still need intake, ownership and classification for the data you send them. The vendor's certifications don't settle your obligations, as your AI vendor's compliance doesn't make you compliant sets out.

  • Low-risk, read-only internal tools over already-public content can treat lineage lightly. Once an output reaches a customer or informs a decision about a person, lineage stops being optional.

Where regulation already expects this

If your feature is a high-risk system under the EU AI Act, much of this is a legal duty, not good practice. Article 10 requires training, validation and testing data to be "subject to data governance and management practices appropriate for the intended purpose", covering among other things "data collection processes and the origin of data", and requires datasets to be "relevant, sufficiently representative, and to the best extent possible, free of errors and complete" (EU AI Act, Art. 10). Article 12 requires high-risk systems to "technically allow for the automatic recording of events (logs) over the lifetime of the system" (Art. 12). After the Digital Omnibus, those high-risk obligations apply from 2 December 2027 for Annex III systems and 2 August 2028 for Annex I systems (Regulation (EU) 2026/1744). If you are not sure whether your system is in scope, start with the high-risk classification guide.

Where personal data is involved, GDPR applies whatever the risk tier. Article 5 sets purpose limitation, data minimisation, accuracy and storage limitation, and makes the controller "responsible for, and be able to demonstrate compliance with" them (GDPR Art. 5). Article 30's record of processing includes, where possible, "the envisaged time limits for erasure of the different categories of data" (Art. 30), and Article 17 gives people the right to erasure "without undue delay" (Art. 17). In our judgement (confirm it with your legal team), an erasure that removes a record from the source but leaves it in an embedding index and months of prompt logs has not been fully met. That is why classification and retention sit before ingest in the table.

For delivery teams the point is practical: the evidence an auditor asks for is what the six controls already produce. Build them as delivery artefacts and the compliance file mostly assembles itself.

Implementation reality: where this breaks

Owners say yes and then don't answer. AI consumers are not their job. Keep the ask small, a one-page record and a 30-minute call, and make the consequence explicit: unowned data stays out of the index.

Contracts become shelfware. A contract not checked on every run is a wish. Put its checks in the pipeline, the part that fires at 02:00 when nobody is reading documents.

Lineage is added after the first incident. Logging chunk identifiers and build manifests from day one is far cheaper than reconstruction. For an existing pipeline, the guide to data pipelines for ML shows where those records naturally fit.

Governance gets run as a separate review. A data review held once at launch catches the state of the data on that day. The support assistant's data was fine on launch day. Put the controls in the pipeline and the definition of done, as described in AI governance in the pipeline, so they run every time the data changes.

What to do next

This week, run the trace-one-answer test on one AI feature in production and write down the score and the missing controls.

Then take the weakest source to its owner, or find out it has none, and fill in the AI Data Intake & Contract template together. If you are at the start of a new AI project instead, begin with the data questions to ask before an AI project starts, and ask them before design, while the answers can still change the plan.

Final takeaway

AI features are evaluated on their models and fail on their data, because data changes without a release and AI systems absorb bad data without complaint. Treat every data source as a dependency with an owner, a promise and a risk: settle intake, ownership and classification at discovery, then enforce contracts, quality gates and lineage on every run. The data is the first quality gate, and it is the one most launch reviews never inspect.

Sources

Frequently Asked Questions

What is data governance for AI delivery?

The set of decisions and controls that make the data under an AI feature fit for purpose and accountable: who owns it, what it promises, how it is checked before use, how answers trace back to it, and how it is classified, retained and deleted. It is placed inside the delivery lifecycle, not run as a separate committee.

Why do AI features that pass their evaluations still fail?

Evaluations are conditional on the data they were run against. If the data in production differs from the data in the eval, because a source changed, an old folder was included or a filter broke, the eval result no longer describes the live system. Data changes on its own clock, without a release.

What is the minimum data governance a small team needs for an AI feature?

For each data source: a named owner, a one-page intake record, a short written contract with the producer, two or three automated checks that block a bad index or training run, a record of which document versions were used, and a classification and retention decision made before ingest.

bottom of page