top of page

Data Quality Gates: Test the Data Like You Test the Code

Shift Quality Editorial
57 minutes ago
11 min read

Every job in the pipeline passed. The assistant still spent nine days answering from documents that had lost most of their text. The pipeline was checking that the work ran. Nothing was checking what the work produced.

(Composite scenario. The details are drawn from patterns that recur across organisations running retrieval-augmented assistants, and are not a single verified incident. Figures are illustrative.)

A mid-sized logistics company ran an internal assistant that answered depot supervisors' questions about operating procedures: dangerous goods handling, vehicle checks, incident reporting. It used retrieval-augmented generation (RAG). A nightly job pulled about 4,200 procedure documents from the document management system, extracted and chunked their text, embedded the chunks and rebuilt the search index the assistant reads from.

One Thursday the document system's export connector was upgraded. The new version handled scanned PDFs differently: for roughly a third of the procedures, text extraction returned the cover page and the revision table and nothing else. Every job in the nightly run reported success. The document count was 4,200, as it was the night before. The new index was swapped in at 02:00.

On Friday a supervisor asked whether a damaged drum of a corrosive substance could be repacked on site. The full procedure said no, with a specific exception for a trained handler and the right overpack. The surviving fragment of that procedure was its one-page summary, which mentioned repacking and not the conditions. The assistant gave a confident, incomplete answer, with a citation to the right document.

The problem surfaced nine days later, when a safety coordinator noticed that answers had become vague. The thumbs-down rate had crept from about 4% to 7%, but nobody had a threshold for that number.

The total chunk count had fallen from about 61,000 to about 38,000 in one night. That number was in the build log the whole time.

Why "the pipeline is green" feels like enough

Teams that build AI features usually test their code well: unit tests on the ingestion job, retries and paging on failed steps, dashboards showing the job ran on time. That instinct is reasonable, because most pipeline failures used to be crashes, and crashes are loud.

Data failures are quiet. The connector upgrade threw no error. It returned valid, well-formed, much shorter text, and every step downstream did its job correctly on bad input. Code tests ask "did the transformation do what we wrote?" Nobody had asked "is what came in still what we expect?"

Google's TFX team made this argument in a 2019 paper on their production data validation system: data in a machine learning pipeline deserves the same attention as the training code, and errors in it are detected far earlier when the pipeline checks the data against an expected schema and statistics rather than waiting for model quality to drop (Breck et al., 2019). The point applies at least as strongly to RAG. A retrieval index is rebuilt more often than most models are retrained, and its failures go straight to users.

The discovery move: at intake, before anyone builds the pipeline, ask the data owner, "What does a bad day of this data look like, and what should the system do when it sees one?" The logistics company could have answered the first half in five minutes: documents with almost no text, a sudden drop in volume, procedures missing from one depot's folder. Nobody asked, so nobody turned the answers into checks. That question belongs with the others in the data questions to ask before an AI project starts.

The quality dimensions that matter for AI

Data quality frameworks list many dimensions. Five do most of the work for AI features, and each needs a meaning specific enough to measure.

Completeness is not "no null fields". For documents, it means the extracted text is a plausible share of the source: measure characters per page against that document's own history. The logistics company's broken documents fell from around 2,500 characters a page to under 200.

Validity means the data conforms to what downstream steps assume: readable encoding, dates that parse, labels from the agreed set, no chunk that is just a repeated page header.

Freshness means the data is as recent as the use case needs. Measure the gap between each source's last modified time and its last indexed time.

Uniqueness means each fact appears once, or duplicates are deliberate. Version 3 and version 4 of the same procedure, both live, make retrieval return contradictory chunks, and the model has to pick one without knowing which is current.

Representativeness means the data covers the situations the system will be used in. For a model, this is the population the training set describes, and where bias hides (see bias in training data: where it hides and how to find it). For a retrieval index, it is coverage: does every depot, product line and procedure category still have documents in the index after the build?

These dimensions line up with the EU AI Act's wording for high-risk systems. Article 10(3) requires that "training, validation and testing data sets shall be relevant, sufficiently representative, and to the best extent possible, free of errors and complete in view of the intended purpose" (Regulation (EU) 2024/1689, Art. 10). Two limits are worth stating plainly. The duty falls on providers of high-risk systems, and for systems that don't involve training models, Article 10(6) says paragraphs 2 to 5 "apply only to the testing data sets". The Act does not mention retrieval indexes by name, so whether your document corpus is in scope is a question for legal advice. For Annex III systems, the high-risk obligations now apply from 2 December 2027 under the Digital Omnibus (Regulation (EU) 2026/1744); the high-risk classification guide helps you work out whether yours is one. Either way, the five dimensions are the right things to test.

To test your own position, pick one AI feature and write down, for each of the five dimensions, the number you would check and where it is recorded today. A dimension with no number is one you are not testing.

Where the gates sit

A gate is an automated check on the data at a fixed point, with an action agreed in advance for when it fails. There are three points where gates pay for themselves, and they check different things because they see different amounts of the data.

The ingest gate sees one record or document at a time, so it checks the item itself: did text extract, is the encoding readable, is there a named owner in the metadata. Its failure action is usually quarantine: set the document aside, load the rest, report what was held back. Failing the whole batch for one corrupt file punishes 4,199 good documents for one bad one.

The build gate sees the whole dataset or index at once: for training, when the training set is assembled; for RAG, after chunking and embedding, before the new index serves anyone. Only here can you check aggregate properties: volume against the previous build, distribution across sources, duplicate rate, chunks with null embeddings. The logistics company's failure was invisible per document (each broken file had some text) and obvious in aggregate (37% fewer chunks overnight).

The pre-release gate checks behaviour, not data. Before a new index replaces the old one, run a fixed set of known questions and check the expected documents come back in the top results; before a retrained model is promoted, run it on a checked held-out test set. This catches problems the data checks weren't designed for, such as a chunking change that splits every table in two.

Diagram of a RAG index pipeline with three data quality gates, an ingest gate that quarantines bad documents, a build gate that blocks the swap on aggregate checks, and a pre-release retrieval check, with yesterday's index kept serving when a gate fails

For RAG, the build and pre-release gates depend on one design decision: build the new index alongside the old one and swap only when the gates pass. If the nightly job overwrites the live index in place, there is nothing to fail closed to, and every gate becomes a warning by default. If your pipeline overwrites in place, change that first; it is a pipeline change, not a new check.

Example checks per gate

The table below is a starting set, not a standard. Thresholds are illustrative: set yours from the history of your own data, ideally as part of the agreement with the data's producer (see data contracts for AI features).

Gate

Example check

Dimension

Default action

Ingest

Extracted characters per page below 30% of that document's previous version

Completeness

Quarantine the document, keep its previous version in the index

Ingest

Text fails to decode, or more than 5% of characters are replacement characters

Validity

Quarantine

Ingest

No owner or classification in the source metadata

Validity

Quarantine (see classify before you ingest)

Ingest

Source modified more than N days after the version last indexed

Freshness

Warn the source owner

Build

Total chunk count changes more than 15% from the last good build

Completeness

Fail closed: keep yesterday's index

Build

Any source folder or category drops to zero documents

Representativeness

Fail closed

Build

Near-duplicate rate above the agreed level, or two live versions of one document

Uniqueness

Warn; fail closed if it doubles

Build

Any chunk with a null or zero embedding vector

Validity

Fail closed

Build (training)

Label distribution shifts beyond the agreed band from the last training set

Representativeness

Fail closed pending review by the data owner

Pre-release

Expected document in the top 5 results for fewer than 90% of a fixed question set

Completeness, validity

Fail closed

Pre-release

Median age of retrieved chunks rises above the freshness target

Freshness

Warn

Run that table against the logistics company's Thursday. The characters-per-page check would have quarantined about 1,400 documents and kept their previous versions serving. Failing that, the chunk-count check would have blocked the swap; failing that, a few dozen known safety questions would have caught the dangerous goods answer. Any one of the three turns a nine-day incident into a morning ticket.

Fail closed, quarantine or warn

Every gate needs its action decided before it fires, because at 02:00 nobody is there to decide. There are three options, and the choice is a trade-off between the cost of serving bad data and the cost of serving no new data.

Action

What happens

Gains

Costs

Best suited for

Warning signs

Fail closed

The pipeline stops; the last good version keeps serving

Bad data never reaches users

Fresh data is delayed; a noisy check blocks good builds

Aggregate checks with a known-good fallback; anything users act on without review

People start disabling the check to get builds through

Quarantine

Bad records are held back; the rest load

The batch still ships; problems are isolated

Someone has to work the quarantine queue

Per-record checks at ingest

The quarantine queue grows and nobody owns it

Warn

The data loads; a named person is alerted

No delay

Bad data reaches users until someone acts

Noisy or early-signal checks; cases where stale is worse than degraded

Warnings nobody reads; the same warning firing every night

Three conditions favour failing closed: the defect would reach users with no human in between, a known-good fallback exists, and the check rarely fires on good data. The logistics company's chunk-count check meets all three. A freshness warning on one document meets none of them.

The exception cuts the other way. For fast-changing data that matters, such as a product recall or sanctions list, yesterday's version can be more dangerous than today's imperfect one. There, a failed gate should page someone at once rather than silently keep the old index, with the fallback agreed with the data owner in advance.

Most data testing tools can express this choice. In dbt, for example, each data test has a severity of error or warn, with warn_if and error_if thresholds on the number of failing rows, so one test can warn above 10 failures and error above 1,000 (dbt documentation). The tool is incidental; the threshold and action must be written down and reviewed, not held in one engineer's head.

A warning is only a control if it has an owner and a deadline. Count the data quality warnings that fired last month, and how many led to an action. If the second number is close to zero, promote the useful warnings to fail closed and delete the rest.

Implementation reality: where gates break

Thresholds come from nowhere. A team picks "15%" because it sounds sensible, and the gate either never fires or fires every Monday when the weekly batch lands. Set the first thresholds from 30 to 60 days of the metric's own history, then review them with the data owner after the first month. Ownership matters here: the person who can say whether a 20% drop is normal is the data owner, not the pipeline engineer (see who owns the data your AI uses).

Gates get switched off under pressure. A gate that blocks a release with no explanation gets bypassed. Every failure message should say which check failed, by how much, which sources are affected and who to call. Bypasses need a named approver and a log entry.

Nobody can tell which build served which answer. Record the build identifier, gate results and source document versions with each build, and log the build identifier with each answer. That record is the start of lineage for AI inputs.

Gates are added to the pipeline but not to the change process. The logistics company's trigger was a connector upgrade. Any upstream change (connectors, extraction libraries, chunking settings, embedding models) should go through a gated build before production, the same rule you apply to code. For the wider set of release controls, see AI governance in the pipeline: a checklist.

What to do next

Pick one AI feature fed by an automatic refresh and find out three things this week: whether the refresh builds alongside the live version or overwrites it; the total record or chunk count for each of the last 30 builds (the normal range is your first threshold); and what its data owner says a bad day looks like.

Then add one gate: the aggregate volume check at build time, failing closed against the previous good build. It is the cheapest gate to write, and it would have caught the logistics company's failure on the first night. Record the threshold, the action and the owner in the AI Data Intake & Contract template, so the next person to change the pipeline knows the gate exists and why.

This post is part of the data governance for AI series. The anchor, Data Governance for AI Delivery, sets out where quality gates sit among the six controls.

Final takeaway

Code tests tell you the pipeline did what you wrote, not that the data going through it was any good. Gate the data itself at ingest, at build and before release, in numbers you can find, with the action for each failure decided in advance. Build the new version alongside the old one, so that failing closed means users keep yesterday's good data instead of today's broken data.

Sources

Frequently Asked Questions

What is a data quality gate?

An automated check on the data itself, at a fixed point in the pipeline, with an action decided in advance for when it fails: stop the pipeline, quarantine the bad records, or let the data through and alert a named person. A check with no agreed action is a dashboard, not a gate.

Where should data quality gates sit for a RAG system?

At three points: per document at ingest (did the text extract properly?), on the whole index at build time (did volume and coverage stay within normal range?), and before the new index replaces the old one (does it still retrieve the right documents for a fixed set of known questions?).

When should a data quality gate fail closed rather than warn?

Fail closed when bad data would reach users with no human in between, a known-good fallback exists (such as yesterday's index), and the check is precise enough that it rarely fires on good data. Warn when the check is noisy or when stopping would leave users with something worse than slightly degraded data.

bottom of page