top of page

Data Contracts for AI Features

Shift Quality Editorial
58 minutes ago
11 min read

Most AI features don't break when someone changes the model. They break when someone upstream changes the data and nobody downstream was told. A data contract is how you make that change impossible to ship quietly.

At a mid-sized logistics company, the last-mile team ran a model that predicted which parcels would miss their delivery window. A high score triggered a proactive customer message and a reroute flag. Its strongest input was a simple one: whether the parcel had picked up an EXCEPTION scan at the depot, meaning it could not proceed on its planned route.

The depot systems team, which owned the scan events table, wanted better reporting, so they split EXCEPTION into EXCEPTION_ADDRESS and EXCEPTION_ACCESS. Their tests passed: the column was still a non-null string and row counts were normal.

The feature pipeline for the delivery model mapped status codes it knew to categories and sent anything else to OTHER. That default had been added years earlier "so the job never fails." From the day of the change, the model's strongest feature was almost always zero. The model did not crash or lower its confidence. It simply stopped seeing exceptions, and its recall on late parcels slid from about 0.7 to about 0.5 over five weeks. The first signal anyone noticed was a rise in complaints from customers whose parcels were late with no warning.

(Composite scenario. The details are drawn from patterns that recur across organisations running models on operational data, and the figures are illustrative, not from a single verified incident.)

Nobody did anything wrong by their own standards. What was missing was any written, checked agreement that the meaning of scan_status was something another team relied on.

Why "we already test our data" feels like enough

The common response to a story like this is to add more tests. But both teams had tests, and both suites passed. The producer tested that their data was valid by their own definition; the consumer, through the OTHER default, tested only that the pipeline completed. Each team tested its own side of the boundary, and nobody tested the boundary itself. A schema check confirms that a column exists and has a type. It cannot tell you that a code's meaning changed, a field moved to UTC, or a table now lands at 07:00 instead of 03:00. Those are promises between teams, and they have to be written down where both can see them and a machine can check them.

For an AI feature the gap is worse than for a report, because a model turns a broken input into a confident output. It doesn't throw an error when a feature goes quiet; it produces a plausible answer from what's left. That's why the anchor post in this series, Data Governance for AI Delivery, treats contracts as one of six controls rather than an optional extra.

Test your own boundary. Pick the AI feature you are most accountable for and list its five most important inputs. For each, ask: if the upstream team changed what this field means tomorrow, which of our checks would fail? If the answer is "none, but we'd notice in the metrics eventually", you are relying on the same luck the logistics company had.

What a data contract actually promises

A data contract is a written, versioned agreement between the team that produces a dataset and the teams that consume it. Most explanations stop at schema, and a schema-only contract would have passed the logistics change without comment. A useful contract makes five promises:

Promise

What it states

What breaks without it

How a machine checks it

Schema

Field names, types, required fields, keys

Pipelines fail loudly, or worse, coerce types silently

Compare the contract with the live table on every run

Meaning

What each field represents, its units, time zone and the full list of valid codes with definitions

Features shift in meaning while every type check passes

Valid-value checks, and a breaking-change rule on code lists and descriptions

Freshness

How late data may arrive and how often it updates

Models score on yesterday's state; retrieval answers from stale documents

Latency and frequency checks against a timestamp field

Quality thresholds

Numbers that count as failure: nulls, duplicates, invalid values, row counts

"Mostly fine" data degrades a model gradually

Threshold rules that fail the run, not just log a warning

Change notice

Which changes need notice, how much, and to whom

Consumers find out from their users

A version bump that blocks the producer's merge until consumers acknowledge

Meaning is the promise most teams skip, and it was the one that failed at the logistics company. Make it testable by treating code lists as part of the contract: scan_status is not a "string" but one of six defined values, and adding a seventh is a change someone downstream has to approve.

The discovery move sits here. The question that would have caught the failure belongs at intake, before the model was built: which of our inputs carry business codes, and who decides when those codes change? That question belongs in the intake conversation described in The Data Questions to Ask Before an AI Project Starts. If the answer is "nobody in particular", the contract is the place to fix it.

How a silent degradation travels

The logistics failure followed a chain that repeats wherever an AI feature consumes data from a team it doesn't control:

Local change → producer tests pass → consumer coerces → feature shifts → model stays confident → detection lags

The producer's change is correct for their own users, and their tests check their own definition of valid. A default, a cast or a fill value in the consumer's pipeline absorbs the new data. The distribution of one input shifts, but no single row looks wrong, and the model has no way to know an input now means something different. Outcome metrics such as recall need ground truth, which for late deliveries arrives days after the prediction, so the drift shows up weeks later, if at all.

Every arrow after the first is a place where a check could stop the chain. A contract moves detection from the lagging metric, where it costs weeks of degraded predictions, to the producer's pull request, where it costs a failed build.

Diagram comparing an upstream status-code change that travels unseen through producer tests, a feature pipeline and a model until customer complaints in week five, with the same change stopped by contract checks on the producer pull request, on every data run and in the consumer build.

A minimal contract, written down

You don't need a platform to start: a YAML file in the producer's repository, reviewed like code, will do. The Open Data Contract Standard (ODCS) gives that file a shared shape. ODCS is maintained by Bitol, a Linux Foundation AI & Data incubation project, under the Apache 2.0 licence. The current release is v3.2.0, published in September 2026 after v3.1.0 in December 2025 (Bitol; ODCS releases).

Here is the contract the depot scan table should have had, cut down to what the delivery model depends on. Field names follow ODCS v3.2.0; the values belong to the composite example.

apiVersion: v3.2.0
kind: DataContract
id: depot-scan-events
name: depot_scan_events
version: 1.4.0            # a new scan_status value means 2.0.0
status: active
domain: last-mile
schema:
  - name: scan_events
    physicalType: table
    properties:
      - name: parcel_id
        logicalType: string
        required: true
        quality:
          - metric: nullValues
            mustBe: 0
      - name: scan_ts
        logicalType: timestamp
        required: true
        description: Time the scanner recorded the event, in UTC.
      - name: scan_status
        logicalType: string
        required: true
        description: >-
          Physical event at the depot. EXCEPTION means the parcel cannot
          proceed on its planned route for any reason.
        quality:
          - metric: invalidValues
            arguments:
              validValues: [RECEIVED, SORTED, LOADED, OUT_FOR_DELIVERY, DELIVERED, EXCEPTION]
            mustBe: 0
slaProperties:
  - property: latency
    value: 1
    unit: d
    element: scan_events.scan_ts
    driver: operational
customProperties:
  - property: changeNoticeDays
    value: 10
  - property: consumers
    value: [late-delivery-model, depot-reporting]

Three details do most of the work. The description on scan_status states what EXCEPTION means, so splitting it becomes a conversation about meaning, not a rename. The invalidValues rule turns the code list into a test that fails the day a new code appears. And the version comment sets the team's rule that adding a code is a major change. That follows Semantic Versioning's definition of a major version as an incompatible change (SemVer 2.0.0). Adding a value looks backward compatible to the producer, but it is incompatible for any consumer that maps codes.

The latency property is ODCS's preferred term for what most teams call freshness. Notice period and the consumer list are not standard ODCS fields, so they go under customProperties. ODCS also has sections for team, roles, support channels and data quality at the table level, which are left out here to keep the example short (ODCS documentation).

Check your own version. Try writing this file for the most important upstream table your AI feature reads. Every field whose meaning you can't fill in without asking someone gives you a name, and that person belongs in the next section.

Who agrees it

A contract nobody has agreed is documentation, and documentation drifts. The value is in the agreement: the producer accepts an obligation, the consumer accepts the limits of what is promised, and both need people with the authority to commit.

Role

Who, in practice

What they agree to

Producer owner

Accountable owner of the source system or table

To meet the promises and give notice before changing them

Consumer owner

Owner of the AI feature, accountable for its outputs

To depend only on what is promised, and to respond to notice within the window

Data steward

Whoever can rule on what a business field means

To settle disputes about meaning, such as whether a new code is "the same thing"

Engineers on both sides

The people who build the pipelines

To draft the contract and wire up the checks (they draft it; they don't sign it)

If the producing table has no accountable owner, nobody can agree the contract, and that is a finding in itself; Who Owns the Data Your AI Uses? covers how to assign one without a reorganisation. Until then, the consumer can enforce a one-sided contract, which protects the model but gives no notice.

Producers often resist contracts as an obligation with nothing in return, so give them something back: the consumer list tells them exactly who a change will affect, before the incident call rather than during it.

How to enforce it in CI

A contract checked only at review time protects you until the first change nobody reviews. Enforcement runs automatically in three places, because each catches failures the others miss.

1. The producer's pull request. When the contract file changes, compare the new version with the previous one and classify the change. Removing a field, narrowing a type, adding or removing a code, or changing a description of meaning is breaking: the build fails unless the major version is bumped and the consumer owners named in the contract have acknowledged it. Adding an optional field is not breaking. Open-source tools can do the comparison; the Data Contract CLI, for example, is MIT-licensed, reads ODCS files, runs as a GitHub Action, and has lint, test, changelog and breaking commands (Data Contract CLI). Treat it as an example: your rule for what counts as breaking, especially for code lists, is a decision you write down, not a default you inherit.

2. Every data run. Some changes never touch the contract file: a vendor reconfigures a scanner, or a backfill writes in the wrong time zone. So the contract's rules also run against the data on every load: valid values, nulls, row counts, latency. At the logistics company this check alone would have failed on the first night EXCEPTION_ADDRESS appeared. Where these gates sit and whether they should fail closed or warn is the subject of Data Quality Gates: Test the Data Like You Test the Code.

3. The consumer's build. The training pipeline, batch scoring job and retrieval index build each record which contract version they were built against, and refuse data from a different major version. Remove silent defaults like OTHER for contracted fields: an unknown code should stop the run. This check protects you when the producer's CI is misconfigured or bypassed.

Recommended threshold (ShiftQuality's recommendation, not an external standard): all three checks fail closed for any input among a model's most influential features or that decides what a retrieval index contains. Elsewhere the data-run check may warn, if someone owns the warning.

The record of which contract version fed which model build is also lineage, and Lineage for AI Inputs covers what else to record alongside it. For how these checks fit into a wider set of release controls, see AI governance in the pipeline: a checklist.

Tradeoffs and where this changes

Option

Gains

Costs

Best suited for

Warning signs

Consumer-side checks only

Fast, needs no one else's agreement

No notice; you find out at run time

Inputs from teams that won't engage yet, third-party feeds

Frequent failed runs that need emergency fixes

Producer-agreed contract, checked in CI

Changes are caught before they ship, with notice

Negotiation time; producers carry an obligation

Internal inputs to production AI features

Contracts with no acknowledgement on the last major change

Central contract platform

Discovery, one catalogue, consistent rules

Licence and platform effort; can become a bottleneck

Organisations with dozens of producer teams

Contracts written to satisfy the platform, not the consumer

Exceptions worth making explicitly:

  • Prototypes. An experiment that may never ship needs consumer-side checks, not a signed contract. Write the contract when the feature is approved for production.

  • Third-party data. A vendor won't sign your YAML. Write the contract anyway, enforce it on your side, and use it to set data terms in the next renewal.

  • Unstructured documents for retrieval. The contract covers what you can promise: required metadata (owner, version, classification), accepted formats and sync frequency.

The tradeoff underneath all three is speed against notice: consumer-side checks are quick but tell you after the fact, while an agreed contract is slower to set up and is the only option that gives you warning.

Implementation reality: where this breaks

Contracts drift from the data. Generate what you can from the live schema, and run the data-run check from day one so a drifting contract fails loudly rather than misleads quietly.

Notice periods collide with emergencies. A producer fixing a production defect cannot wait ten days. Define an emergency path: the change ships, consumers hear the same day, and the consumer's build check protects the model meanwhile.

Acknowledgement becomes a rubber stamp. Make the acknowledgement name the action taken: mapping updated, retraining scheduled, or no impact with a reason.

Regulated systems need evidence. For high-risk AI systems, Article 10 of the EU AI Act requires data governance practices covering, among other things, data origin, data-preparation operations and "the formulation of assumptions" about what the data represents (EU AI Act, Art. 10). A contract doesn't make a system compliant, but it is a versioned record of exactly those assumptions and their checks. If you aren't sure whether your feature falls into that category, start with High-risk AI systems under the EU AI Act.

What to do next

This week, pick one AI feature and its most influential upstream input, and write the contract for that one table with all five promises. Every field you can't complete without asking someone is a gap in your intake, not in your YAML.

Then get it agreed by the producer owner, the consumer owner and a steward, and turn on the data-run check before anything else. It needs no one's permission and would have caught the logistics failure on night one. The AI Data Intake & Contract template gives you the intake questions and a contract page to record the agreement.

Final takeaway

AI features fail quietly when the data under them changes meaning and nobody downstream is told. A data contract makes five promises (schema, meaning, freshness, quality thresholds and change notice), agreed by the people who own both sides of the boundary, and checked by machines on the producer's pull request, on every data run and in the consumer's build. Write the first one for the input your model depends on most.

Sources

Frequently Asked Questions

What is a data contract?

A written, versioned agreement between the team that produces a dataset and the teams that consume it. It states the schema, what each field means, how fresh the data will be, the quality thresholds it must meet, and how much notice consumers get before any of that changes. It is useful only when machines check it.

Who should agree a data contract for an AI feature?

The accountable owner of the producing system, the owner of the AI feature that consumes the data, and the data steward who can rule on what a field means. Engineers on both sides draft it; those three people agree it and own the change notice.

How do you enforce a data contract in CI?

In three places: a breaking-change check on the producer's pull request, contract tests run against the data on every load, and a check in the consumer's training run or index build that refuses data from a contract version it was not built against.

bottom of page