top of page

Classify and Set Retention Before You Ingest Data Into RAG

Shift Quality Editorial
57 minutes ago
10 min read

A retrieval index is a copy of your documents that has forgotten who was allowed to read them and when they were meant to be deleted. Both decisions have to be made before the copy exists.

The assistant had been live for five weeks when a junior analyst asked it a reasonable question: "What is the salary band for my grade, and how does my pay compare?"

The answer was helpful and specific. It quoted the band, then added three lines from a spreadsheet called Comp review FY final, listing the salaries of four named people on the analyst's team. The analyst screenshotted it. By lunchtime the HR director knew.

The organisation was a mid-sized professional services firm. The assistant was an internal "ask HR" tool built on retrieval-augmented generation (RAG): documents are split into chunks, embedded and stored in a vector index, and the closest chunks are handed to the model as context at question time. The delivery team had pointed the ingest job at the HR shared drive using a service account that could read everything, because that gave the best coverage. The compensation folder was restricted to HR and Finance. The index did not know that.

The clean-up took longer than the incident. The spreadsheet was removed from the drive that afternoon, but the nightly sync only added and updated documents, so its chunks stayed in the index. The answer cache still held the salary reply. The observability tool had logged every prompt and response for 90 days, readable by the platform team. It took eleven days to confirm every copy was gone.

(Composite scenario, drawn from patterns that recur in internal assistant rollouts; not a single verified incident.)

The model did nothing wrong. The failure was a data decision nobody made: what this content was, who could see it, and how long it could live in each copy, settled before the first chunk was written.

Why "index first, lock down later" is attractive

First RAG assistants are judged on answer quality, and coverage drives answer quality. Three arguments make broad ingest feel safe. The source systems already have permissions, refined over years. The assistant is internal, so every user is an employee. And classification sounds like a quarter's work when the pilot is due in three weeks. All three fail on the same mechanism.

What the common view misses: an index is a copy, and copies do not inherit

Ingest takes text out of the system that enforced its permissions and puts it somewhere new. The new store knows only what you copy into it. Microsoft's documentation for its search service is candid about this: in the document-filtering pattern, "There's no authentication or authorization through the security principal. The principal is just a string, used in a filter expression" (Microsoft Learn, 2026). That product is only an example; the pattern holds across vector stores. Permission data in an index is a snapshot written at ingest, enforced only if every query applies it.

Four consequences follow, and each maps to the firm's incident:

  1. The ingest identity becomes everyone's identity. Whatever the service account can read, every user can retrieve.

  2. Permissions drift. A person removed from a group at the source keeps access in the index until the next sync rewrites that document.

  3. Embeddings are not a safe derivative. Research at EMNLP 2023 recovered 92% of 32-token text inputs exactly from their embeddings (Morris et al., 2023), and OWASP warns that attackers can "invert embeddings and recover significant amounts of source information" (OWASP LLM08:2025). Classify the vector at the level of its text.

  4. Every answer makes more copies. Retrieved chunks flow into prompts, responses, the answer cache, trace logs and evaluation sets, each with its own readers and retention.

OWASP's mitigations read like a pre-ingest checklist: "permission-aware vector and embedding stores," and "Tag and classify data within the knowledge base to control access levels" (same source). Both are ingest-time decisions. You cannot tag chunks after their metadata has been thrown away.

One source document copied into five places after RAG ingest (chunk index, embeddings, answer cache, logs, evaluation sets), with the three decisions that must travel with it: classification tier, access rule and deletion key.

Test it now: for one document in your current assistant, list every place its text or vectors live and who can read each. Fewer than four entries usually means you have not found them all.

Decision 1: classify at the source, and fail closed

You do not need a new scheme. Most information security policies already have three or four tiers. They need to become ingest rules, applied per folder or document, with one default: unclassified content is not ingested.

Tier

Typical content

Ingest rule

Access in the index

Derived copies (cache, logs, evals)

Public

Published policies, product docs

Ingest

All users

Follow source

Internal

Process guides, internal FAQs

Ingest

All employees

Standard log retention

Confidential

Team plans, contracts, customer data

Ingest with access list copied and synced

Per-document filter, parity-tested

Redacted or short-lived logs; no answer cache

Restricted

Pay, health, disciplinary, special category data

Exclude by default; separate owner-approved index if needed

Separate index per audience

Never log content; deletion drill before go-live

The tiers are ShiftQuality's recommendation, not a standard; use your policy's names. The last column is the one nobody fills in, and it is where a well-protected confidential document ends up in plain text in a trace log forty engineers can read.

The discovery move: before ingest, ask the source owner one question per folder: "Which tier is this, and what happens if every employee can read it?" Asked of the HR drive owner, it surfaces the compensation folder in a minute. If nobody can answer, the source has no owner, which is the first problem to solve; see who owns the data your AI uses.

The test: count the documents in your ingest manifest with no tier. That number is the data whose exposure you have decided by accident.

Decision 2: carry access controls into the index, and prove it

Option

Gains

Costs

Warning signs

Exclude restricted sources

Nothing to leak

Lower coverage

Users paste restricted documents into chat instead

Partition: one index per audience

Clear boundary

Duplicated content, more indexes

Partitions multiply; people added to the wrong one

Filter on copied ACLs per chunk

Fine-grained, one index

Sync lag, large group lists

No measured lag; filter optional in app code

Check at query time against the source

Always current

Latency, source API limits

Checks silently skipped on timeout

Whichever you choose, the chunk schema must carry the source document ID and the access data you will filter on. Adding either later means re-ingesting everything, which is why this is a pre-ingest decision.

Then prove it with a permission-parity test, the RAG version of a negative test. Pick twenty confidential and restricted documents and write a query that should retrieve each. Run it as a user who cannot open the source: expect zero retrievals. Run it as one who can: expect a hit. Remove that user from the source group and time how long until retrieval stops. That is your permission drift window; agree with the data owner whether it is acceptable. These checks belong with your other data quality gates for AI, in the index build pipeline.

Decision 3: set retention and deletion for every copy

For personal data, retention is a legal duty, not an operations detail. The GDPR requires data to be kept in identifiable form "for no longer than is necessary for the purposes for which the personal data are processed" (GDPR Art. 5(1)(e)), and data protection by default covers "the period of their storage and their accessibility" (Art. 25(2)). Article 17 obliges the controller "to erase personal data without undue delay" where a ground in 17(1) applies, with exceptions in 17(3) including legal obligations and legal claims (Art. 17). The UK regulator says to respond "at the latest within one month," that erasure reaches backups too, and that backup data that cannot be overwritten at once must be put "beyond use" (ICO; the ICO marks this guidance as under review following the Data (Use and Access) Act).

The delivery consequence: every copy the assistant makes must be findable by source document within that month. You cannot find content by reading vectors, only by a key written at ingest. So decide three things first:

  • A deletion key on every derived record. Chunks, cache entries, log records and evaluation rows carry the source document ID. It is the same record that lets you trace an answer to its source, covered in lineage for AI inputs.

  • A retention period per copy, per tier. The index follows the source. The answer cache is shorter than your erasure deadline or off above internal. Logs exclude or redact content above internal, or have a retention you can actually search and purge.

  • Sync that deletes. The job must process removals and permission changes, not only additions. An add-only sync is how the firm's spreadsheet outlived its deletion.

The test is a deletion drill. Ingest a canary document containing a unique string, query it so it reaches the cache and logs, delete it at source, and time how long until the string is gone from the index, cache, logs and any exported evaluation set. Any copy that still holds it is a copy your erasure process does not reach.

The firm's assistant, rebuilt with three decisions made first

(Continuing the composite.) The HR drive owner tiers the top-level folders in one meeting. Handbook and policies are internal; recruitment and team plans are confidential; compensation, disciplinary and occupational health are restricted and excluded from the all-staff assistant. The reward team gets its own index, with a named owner signing off its scope.

Every chunk now carries source document ID, version, tier and the drive's allowed groups. The sync processes deletions and permission changes, and the parity test runs after each nightly build. The measured drift window is under 24 hours, which the HR director accepts for confidential content. The answer cache is off above internal; the observability tool redacts confidential content and keeps 30 days instead of 90, inside the one-month erasure window. The deletion drill passes in under a day.

None of this needed new technology. It needed the data owner in the room before the ingest job ran, which is the argument of data governance for AI delivery: data decisions belong at intake.

The pre-ingest checklist

Run it per source (a drive, library, wiki space or database extract) before first ingest and whenever scope changes. A source ships when every line has evidence and a name.

#

Check

Evidence

Signs

1

Source has a named owner with authority over access

Source register entry

Data owner

2

Every folder has a tier; unlabelled count is zero

Ingest manifest

Data owner

3

Restricted content excluded or isolated

Manifest exclusions or separate index

Data owner, security

4

Ingest identity dedicated and scoped to approved paths

Permissions export

Platform owner

5

Chunks carry source ID, version, tier and access data

Index schema

Delivery lead

6

Query filter enforced centrally, not optional in app code

Gateway or code path

Delivery lead

7

Parity test passes; drift window measured and accepted

Test report

Delivery lead, data owner

8

Sync handles deletions and permission changes

Test removal

Delivery lead

9

Retention set per tier for index, cache, logs, eval sets

Retention table

Data owner, privacy

10

Deletion drill passes within target

Timings per copy

Privacy

Lines 2, 5 and 9 are the expensive ones to fix later, because each forces a full re-index. The AI Data Handling Policy template gives you policy wording for tiers and retention, and the AI Data Intake & Contract template records the per-source answers. If you want these questions asked earlier, at project intake, start with the data questions to ask before an AI project starts.

Tradeoffs and where this changes

Failing closed costs coverage: the pilot's answers will be worse at first. Name that to the sponsor as a choice between a narrow assistant that is safe to widen and a broad one whose exposure nobody has measured.

  • Public-only corpora need the light version: tier everything public, still key chunks to source IDs so outdated content can be removed, and skip the parity test.

  • Teams without a data office can classify at folder level with the owner in one sitting, exclude what is unclear, and revisit quarterly.

  • Legal holds. Article 17(3)'s exemptions are a reason to keep the source record, not an answer cache or trace log. Agree with legal which copy is the record.

  • Vendor connectors that claim to carry permissions across: run the parity test against the claim. See why your AI vendor's compliance doesn't make you compliant.

Implementation reality

Labels are missing. Folder-level tiering by the owner is faster and more honest than a classifier run across everything, though a classifier can usefully flag pay or health terms inside folders tiered internal.

Group lists are messy. Nested groups and "everyone except external" entries surface when you copy access lists. Flatten them at ingest, and treat a document shared with a very large group as internal in the index, whatever its label says.

Logs belong to another team. Platform engineering sets trace retention for debugging, not privacy. Agree AI trace rules with them before go-live; logs are the copy most often missed in erasure requests.

Incidents still happen. The deletion key and parity test are what make the response fast. Write the clean-up steps into your AI incident response plan before you need them.

What to do next

Take the retrieval feature you are closest to shipping. This week, run the copy count for one of its sources, then run the deletion drill with a single canary document and time it per copy. If either turns up a copy you did not know about, or a time over a month, stop adding sources until checklist lines 5, 8 and 9 are true.

Final takeaway

Retrieval quietly removes two controls that source systems give you for free: who may read a document, and when it must go. Neither survives the copy unless you write it into the copy. Decide the tier, the access data and the deletion key per source, before the first chunk is embedded, or pay later for a full re-index and a clean-up across systems you did not know held the data.

Sources

Frequently Asked Questions

Do RAG indexes inherit permissions from the source system?

Not automatically. Most retrieval stores hold whatever permission data you copy into them at ingest time, and they enforce it only if every query applies a filter. If ingest runs under a broad service account and no permission data is copied, every user of the assistant can retrieve every chunk.

How do I delete personal data from a RAG system?

Key every chunk, cache entry and log record to its source document ID before ingest. A deletion then becomes a lookup by that ID across the index, the answer cache, the logs and any evaluation sets, rather than a search through embeddings for content you can no longer read.

Are embeddings anonymised data?

Do not treat them that way. Published research has recovered most short text inputs exactly from their embeddings, and OWASP lists embedding inversion as a known weakness. Classify and retain embeddings at the same level as the text they came from.

bottom of page