top of page

Lineage for AI Inputs: Trace Any Answer Back to Its Source

Shift Quality Editorial
57 minutes ago
11 min read

When an AI answer is wrong, the first question is which document it came from. If nobody recorded that at the moment of the answer, the question turns into an investigation, and the investigation runs on memory.

On a Monday morning, a complaint reached the customer relations team at a mid-sized energy retailer. A customer had switched supplier on the strength of the company's support assistant telling her there was no exit fee on her fixed tariff. There was an exit fee. She had the chat transcript, and she wanted the fee waived and an explanation.

The assistant was a retrieval-augmented system: on each question it searched an index built nightly from the company's policy and tariff documents, passed the best-matching passages to a language model, and the model wrote the answer. Waiving one fee was easy. The hard question came from the head of customer operations: how many other customers had been told the same thing, and why?

It took seven working days to answer. (Composite scenario: the details are drawn from patterns that recur in retrieval deployments and are not a single verified incident.)

The application log held the question and the answer, but not which passages had been retrieved. The document library kept only current versions, and the index was rebuilt in place every night, so last month's index no longer existed. Two engineers replayed the question against a reconstructed index, guessing at what it had held three weeks earlier. The cause, when they found it: a superseded "fixed tariff FAQ" from an older product range had been copied into a second folder during a site migration, and the index ingested both. Nobody knew who owned the copy. The person who had moved it had left.

Nothing in that story is a model failure. The model summarised the passage it was given, accurately. The organisation could not see the path from an answer back to its source, so it could not size the damage, fix the source, or show anyone that it had.

Why "we log the prompts" feels like enough

Most teams that ship a retrieval assistant keep logs of the question, the answer, a timestamp and the model name. That feels like an audit trail, and it is cheap: it falls out of almost every framework by default and needs no agreement with the people who own the documents. Lineage, by contrast, asks the ingestion pipeline, the index, the application and the content owners to agree on identifiers. That is a cross-team conversation, and the retailer never had it because nothing forced it.

The trouble is that a prompt log records the two ends of the chain and nothing in between. Every question after a bad answer is about the middle.

What the prompt log misses: four hops, and each one can break

Lineage for an AI input is a chain of four hops. Each hop is a join between two records, and each join needs an identifier that survives over time.

  1. Answer to retrieval event. A request ID links the answer to a record of what the retriever returned, plus the model and prompt-template versions.

  2. Retrieval event to chunk. The retrieval record holds the chunk IDs returned, their rank, and the index build they came from.

  3. Chunk to document version. Each chunk carries its document ID and a version identifier, ideally a hash of the ingested content.

  4. Document version to source and owner. Each document carries its source location, classification and a named accountable owner.

The chain breaks wherever an identifier is overwritten instead of versioned. At the retailer it broke three times: chunk IDs were not logged per answer (hop 2), the library and index kept only current state (hop 3), and "owner" meant "last person to upload", who had gone (hop 4).

This is a discovery failure, not an engineering one. Nobody had asked at intake: if this assistant says something wrong, how will we find the document behind it? That question belongs with source, owner and permission in the data questions to ask before an AI project starts. Asked before the first index build, it costs a schema decision. Asked after a complaint, it costs a week.

Diagram tracing an AI answer back through four hops, from the answer to the retrieval event, the retrieved chunk, the document version and the source and owner, with the identifier to record at each hop and the reverse path used for erasure requests.

What to record at each hop

The table below is the working answer. It is ShiftQuality's recommendation, not an external standard, and it is deliberately minimal: identifiers and hashes, not copies of content.

Hop

Record

Key fields

Who writes it

Keep it for

1. Answer

Response record

Request ID, case ID, timestamp, model version, prompt-template version, answer hash, citations shown

Application, at request time

Your complaint and regulatory window, not the log-storage default

2. Retrieval event

Retrieval record, keyed on request ID

Rewritten query, filters applied (including access filters), chunk IDs, scores, rank, which chunks reached the prompt, index build ID

Application, at request time

As the response record

3. Chunk

Chunk manifest, per index build

Chunk ID, document ID, version hash, position, chunker version, embedding model version, ingest run ID

Ingestion pipeline, at build

Life of any build you may need to reconstruct

4. Document version

Document register

Document ID, version hash, source path, ingested-at, classification, retention rule, named owner, superseded-by link

Ingestion pipeline plus owner

Life of the document plus retention rule

Three design choices in that table do most of the work.

Hash the content you ingested. A hash computed at ingest proves which text the model could have seen; a "last modified" date proves only that someone saved a file. At the retailer, hashes would have shown at once that two different texts both claimed to be the fixed-tariff FAQ.

Give every index build an ID and keep its manifest. You do not need to keep the whole index, only the manifest of which chunks, from which document versions, were in build 2026-09-14-0200. Then "what did the index hold three weeks ago?" is a query, not a reconstruction.

Log which chunks reached the prompt, not just which were retrieved. Many pipelines retrieve twenty candidates, re-rank them and pass five to the model. The five shaped the answer.

To test your system, fill the four rows for one answer from yesterday, from records alone. Every cell you fill by asking a person is a hop that will break in an incident.

Where the existing standards fit

OpenLineage describes itself as "an open framework for data lineage collection and analysis" and is a Graduate project of the LF AI & Data Foundation (OpenLineage; OpenLineage docs). Its object model is built on datasets, jobs and runs, extended with facets, which its documentation describes as "an atomic piece of metadata attached to one of the core entities." That maps well onto hops 3 and 4: the ingest job, its runs, the documents read and the index written. Our reading is that the per-request hop is a different shape of problem: which chunks one customer's question retrieved is an application event, written at request time to your own logs and keyed so it joins to the pipeline lineage. Treat the tool as an example, not a requirement. What matters is that the identifiers join.

NIST's Generative AI Profile (NIST AI 600-1, July 2024) suggests organisations "establish known assumptions and practices for determining data origin and content lineage, for documentation and evaluation purposes" (MP-2.1-001) and test "data and content flows within the GAI system" (MP-2.1-002) (NIST, 2024). These are voluntary, but they give you language a risk committee will recognise.

The EU AI Act requires high-risk AI systems to "technically allow for the automatic recording of events (logs) over the lifetime of the system", supporting traceability appropriate to their intended purpose (EU AI Act, Art. 12). A support assistant like the retailer's is unlikely to be high-risk in itself; check yours with the high-risk classification guide. Either way, the regulator's idea of traceability, events recorded automatically over the lifetime, is the opposite of an index rebuilt in place every night.

The other direction: erasure requests run the chain backwards

Incidents trace forward from an answer to a source. Deletion requests run the other way: from a person or a document to every place its content now sits.

Under the GDPR, a controller must erase personal data "without undue delay" when one of the Article 17 grounds applies (GDPR, Art. 17), and must respond to a request under Articles 15 to 22 "without undue delay and in any event within one month of receipt of the request", extendable by two further months for complex cases (GDPR, Art. 12(3)). Whether a particular request obliges you to erase is a legal question for your data protection officer. Whether you can erase, within a month, is a lineage question.

Content from one source document sits in the document itself, every chunk derived from it, those chunks' embeddings in every live index build, cached answers that quoted it, logs that captured its text, and any evaluation sets built from real conversations. Without a reverse index from document ID to all of those, you are back to searching by hand.

Here lineage and retention pull against each other: lineage wants records kept, erasure wants content gone. The resolution is in the table above. Record identifiers and hashes in the lineage trail, not copies of the text. Erasing a document then means deleting its content, chunks and embeddings and writing a deletion record against its ID, while the trail still shows that version X existed, was ingested by run Y and was erased on date Z. Logs that do hold answer text or passages are copies of personal data and need the same retention rule. Whether a hash derived from personal data is itself personal data is a question for your data protection officer, not an assumption to make. The decisions about classification and retention belong before ingest, which is the subject of classify and set retention before you ingest data into RAG, and the wider privacy picture is in privacy in machine learning.

The reverse test: pick one source document at random and list every index, cache, log and dataset that holds content from it. Time it.

The developed example, replayed with lineage

Here is the retailer's complaint again, with the four records in place. (Continuing the composite.)

The chat has request ID r-81f2. Its retrieval record shows five chunks reached the prompt from index build 2026-09-14-0200. The top chunk, c-30417, maps in that build's manifest to fixed-tariff-faq, version hash 9be1…. The register shows two documents with that title: 9be1… in /legacy-products/, ingested by the August migration run, owner "migration-service"; and 4c07… in /tariffs/current/, owned by the head of pricing.

That is the whole diagnosis, in perhaps fifteen minutes. How many other customers? becomes a query: every response in the last N weeks whose prompt included a chunk from 9be1…. The fix is a register change, not a hunt: remove the legacy copy, rebuild, and add an ingest rule that holds any second document with the same title and a different hash until a person resolves it. That rule is a data quality gate.

The gap that remains is the owner field. "migration-service" is a process, not a person: it tells you where the document is, not who can decide whether it should be there. That is the sibling problem in who owns the data your AI uses.

Tradeoffs: how much lineage, and where it changes

Option

Gains

Costs

Best suited for

Warning signs

IDs and hashes only (default)

Small, cheap to keep, erasure-friendly

Need source or manifests to see the text

Most retrieval assistants

Manifests deleted with old builds

IDs plus passage text in logs

Fast diagnosis

Logs become a second copy of the corpus

Short debugging windows; non-personal corpora

Restricted content in general log stores

Full index snapshots

Exact replay of any answer

Storage; every snapshot is in scope for erasure

Regulated decisions where replay is evidence

Snapshots with no retention rule

Exceptions worth making explicitly:

  • Small, static corpora. A forty-document handbook that changes quarterly can start with versioned documents and build manifests, adding per-request retrieval logs when the corpus or the stakes grow.

  • Live data reads. If the assistant reads a live system, such as an account balance, the "document version" for that hop is the system, the query and the time of the read.

  • Vendor-hosted retrieval. Ask in procurement whether you can get the chunk IDs and document versions behind each answer. If not, record the gap; see why your AI vendor's compliance doesn't make you compliant.

Method: build the chain in three weeks

Sized for a delivery lead to sponsor, with one application engineer, one pipeline engineer and the owners of the largest source collections.

Week 1 — Measure the gap. Forward drill: take five random answers from last week and try to reach the document version and a named owner from records alone. Reverse drill: take two source documents and list every place their content sits. Time each, and note every point where you had to ask a person.

Week 2 — Add the identifiers. In ingestion, hash each document, assign index build IDs and write a chunk manifest per build. In the application, write a retrieval record per request and add model and template versions to the response record. If your pipeline already emits OpenLineage or similar events, attach build IDs and hashes there rather than inventing a second store.

Week 3 — Close the loop. Make a named owner a field the ingest job checks: no owner, no ingest. Add the reverse lookup from document ID to chunks, builds, caches and logs. Set retention for each record type in writing. Re-run both drills.

Recommended threshold (ShiftQuality's recommendation, not an external standard): each drill item in under 30 minutes, from records alone. That puts the answer inside the first hour of an incident and well inside a one-month erasure window.

Implementation reality: where this breaks

Re-chunking breaks old IDs. Change the splitter and every chunk ID changes. Keep chunking method and version in the manifest, and never reuse a chunk ID for different text. A converter upgrade that changes whitespace will also produce new hashes for the same words; accept that, because a false "new version" costs storage and a missed real change costs an incident.

Owners leave. The owner field decays exactly as it did at the retailer. Tie it to the joiner-mover-leaver process and make "owner left" block ingest for that collection.

Logs are split across teams. Response logs belong to the application team, manifests to the data team and the register to content owners. Agree in advance where the join query lives and who can run it, and put it in the runbook; AI incident response covers the rest of that first hour.

What to do next

This week, run the forward drill on five real answers and note the first hop where you had to ask someone instead of querying something. Start there.

Then add the lineage question to intake for the next AI feature: if this says something wrong, how will we find the source, the version and the owner? The AI Data Intake & Contract template has a place for the answer. For how lineage fits with intake, ownership, contracts, quality gates and classification, start from data governance for AI delivery.

Final takeaway

A retrieval assistant is only as accountable as its path back to its sources. That path holds only if every hop records an identifier that is versioned rather than overwritten. Decide those identifiers at intake, keep them as IDs and hashes rather than copies of content, and prove them with a timed drill in both directions.

Sources

Frequently Asked Questions

What is lineage for AI inputs?

The recorded chain that links an AI output back to its sources: the retrieval event that produced it, the chunks retrieved, the exact document version each chunk came from, and the source system and accountable owner of that document. It lets you answer "where did this answer come from?" from a record rather than by investigation.

Does OpenLineage solve lineage for RAG answers?

OpenLineage is an open standard for collecting lineage metadata about datasets, jobs and runs, which suits the ingestion and index-build side of a retrieval pipeline. The per-answer link, which chunks a given request retrieved, is normally written by your application at request time.

How do I know if our lineage is good enough?

Pick a random answer from last week and time how long it takes to name the document version behind it and that document's owner. Then pick a random source document and list every index, cache and log that holds content from it. If either takes more than 30 minutes, the lineage is not usable in an incident or an erasure request.

bottom of page