The Data Questions to Ask Before an AI Project Starts
Most AI projects discover their data problems after the plan, the budget and the date are fixed. Ask the questions at intake, while a bad answer can still change the plan instead of breaking it.
A regional insurer wanted a model to sort incoming property and motor claims into three lanes: fast-track, standard handling, and referral to the special investigations unit. The plan was to train on five years of closed claims, adjuster notes and fraud referral flags from the claims system. Eight weeks to a pilot. The head of claims had budget and a date.
(Composite scenario. The details are drawn from patterns that recur in regulated organisations, and are not a single verified project.)
Before the build was approved, the delivery lead ran a ninety-minute data intake session with the sponsor, the claims platform owner, the head of the investigations unit and someone from the data protection office. Most questions got clean answers. Three did not, and each changed the plan:
"What does the fraud flag actually record?" That an adjuster referred a claim, not that it was fraudulent. Investigation outcomes lived in the unit's own case system, and claims from before a 2022 migration could not be joined to them.
"Were these records collected for this use, and what is in the free text?" The adjuster notes held injury details and information about third parties. The data protection officer wanted an impact assessment before any of it was used for training.
"What will change in these fields in the next twelve months?" The claims platform was being replaced the following spring, and the loss-type codes would change with it.
None of these was a modelling problem. Each would have surfaced eventually: the first in evaluation, as a model that faithfully reproduced adjusters' hunches; the second in a privacy review that stopped the pilot; the third months after go-live, as a model quietly scoring claims on fields that no longer meant what they used to. Found at intake, each cost a conversation. Found later, each would have cost the project.
Why the data questions usually come too late
The usual AI project puts data in phase two. Phase one is the business case: problem, benefit, sponsor, rough architecture, date. Data preparation starts after approval, when engineers pull extracts.
The cost is that every commitment rests on unchecked assumptions. The business case assumes the labels mean what their names say. The timeline assumes the data can be used. The architecture assumes the source will stay still. When engineers find otherwise in week three, the finding arrives as a delay against a committed plan, and the pressure is to work around it rather than redesign.
The room is the second reason. A business-case meeting has the sponsor and the delivery team. The people who could answer the data questions (each source owner, whoever created the labels, the privacy function) are often not invited until something goes wrong. The sponsor answers for them, in good faith and usually optimistically.
The discovery move: treat the data questions as part of the approval decision, not part of the build. A project is not approved until the people who own the data have answered for it, in writing.
A test for your own portfolio this week: take the last three approved AI projects and find the date each data source owner was first named in a project document. If it is after the approval date, the project was approved on unverified data.
What each question is testing
Every intake question tests one of three things about the plan, and knowing which tells you what a bad answer costs:
Viability. Can this data support the outcome at all? A label that records the wrong thing, or a population missing the cases you care about, cannot be fixed by better modelling.
Permission. Are we allowed to use this data for this purpose? The answer can be "yes, with conditions" or "not this data".
Durability. Will the data keep meaning what it means today? A model is a bet that the future looks like the training data, and an upstream change can break that bet without an error message.
Regulators have converged on the same ground. For high-risk systems, Article 10 of the EU AI Act requires data governance covering the origin of data and, for personal data, "the original purpose of the data collection"; "the formulation of assumptions, in particular with respect to the information that the data are supposed to measure and represent"; and "an assessment of the availability, quantity and suitability of the data sets that are needed" (EU AI Act, Art. 10). That is permission, viability and availability, almost word for word. It applies to Annex III high-risk systems from 2 December 2027; whether you are in scope is a separate decision, covered in the EU AI Act classification guide. The voluntary NIST AI RMF asks for the same thinking under MAP 2.3, data collection and selection "(e.g., availability, representativeness, suitability)" (NIST AI RMF Playbook). Even if neither applies, an auditor or incident review will ask these questions afterwards.
The intake question set
Twelve questions across the seven things that decide viability: source, owner, permission, quality, freshness, sensitivity and change. The third column turns a questionnaire into a decision.
Question | Why it matters | What a bad answer means for the plan |
Source. Which systems would the data come from, and can we see a sample this week? | A project that cannot name its sources is not scoped | Not ready for approval. Back to discovery |
Source. For each label: what event created it, and who recorded it? | Labels often record a past decision, not the outcome you want to predict | The model learns to repeat the old decision. Re-scope the target or find the true outcome |
Source. What population does the data cover, and what is missing? | Cases that never reached the system are invisible to the model | Results won't hold for the missing groups. Narrow scope or collect the gap |
Owner. Who is the named owner of each source, and did they agree to this use? | Without an owner, nobody approves access or warns you of change | Every later answer is unreliable. Assign an owner first |
Permission. Why was this data collected, and does that purpose cover training or retrieval? | GDPR requires "specified, explicit and legitimate purposes" (Art. 5) | Privacy review may stop the project late. Get a ruling now or drop the fields |
Permission. Is any of it licensed, third-party or contractually restricted? | Licences and sharing agreements often limit secondary use | Legal exposure. Exclude or renegotiate before build |
Quality. How complete and consistent are the fields we need, measured on a sample? | Owners describe how a system should be used, not how it is | Effort and accuracy forecasts are fiction. Profile a sample before dating the plan |
Freshness. How old is the newest record, and how fast does the world it describes change? | A model trained on stale data predicts a world that has moved on | Accuracy decays after launch. Budget retraining and drift monitoring |
Sensitivity. Is there special category, financial or children's data, or free text that might hold it? | Free text hides what structured fields don't | Impact assessment, access controls, retention rules. Plan them or exclude the text |
Sensitivity. Who may see this data today, and will the AI's users inherit that restriction? | A model or index can show data to people the source would block | Access leak. Design permissions in or narrow the corpus |
Change. What changes to these systems or fields are planned in the next twelve months? | Schema and meaning changes break models silently | Agree change notice with the owner, or plan for a silent failure |
Change. If this data were wrong or had to be deleted tomorrow, how would we find and fix it? | Tests whether lineage, correction and deletion are possible | Make it a design requirement, not a later task |
Record the answers, and who gave them, in the AI Data Intake & Contract template. An answer given verbally by a sponsor is different from one signed by the data owner, and the template keeps that difference visible.
The claims-triage intake, walked through
Here is how the insurer's three bad answers became decisions.
The label. The plan treated "referred to investigations" as "suspected fraud". The head of the unit explained that the flag recorded an adjuster's judgement, made under time pressure, that a claim deserved a second look, and many referrals closed with no finding. A model trained on it would learn which claims adjusters tended to refer, habits and blind spots included. That is how historical data carries old bias into a new system; bias in training data covers how to find it. The true outcome existed in the investigations system, joinable only from 2022. So the fraud lane left release one. The pilot became a two-lane model (fast-track or standard) trained on well-recorded settlement outcomes, and the fraud lane moved to phase two, trained on post-migration investigation outcomes with an owner from the unit named against the label.
The permission. The data protection officer was comfortable with structured claim fields being used to route claims, subject to normal review. The notes were different: injury descriptions are health data, and they named other drivers and witnesses who had never been customers. GDPR requires an impact assessment before processing "likely to result in a high risk to the rights and freedoms of natural persons" (GDPR, Art. 35), and the officer judged that training on the notes probably met that bar. Release one used structured fields only, and the assessment was scheduled ahead of any work on the notes, in the phase-two timeline rather than the pilot's. Privacy in machine learning covers the techniques for when the notes come in.
The change. New loss-type codes would arrive with the replacement platform. Nothing would error, because the new values would be valid strings; the model would just score claims on features it had never seen. The plan gained a mapping layer between source fields and model features, owned by the platform team, and a written agreement that changes to mapped fields come with notice and a test run. That agreement is a data contract, the subject of data contracts for AI features.
The pilot shipped later than eight weeks, narrower, with a phase two that was planned rather than discovered.
How to grade the answers
A question set only helps if a bad answer has consequences. Grade each answer:
Grade | What it looks like | What it allows |
Documented | The owner answered, with evidence: a sample profile, a written ruling, a schema | Proceed |
Verbal | Someone credible answered, no evidence yet | Proceed, evidence due before build starts |
Unknown | Nobody in the room could answer | Time-boxed spike, before approval |
Bad | The answer is known and undermines the plan | Reshape the scope, or stop |
Recommended rule (ShiftQuality's recommendation, not an external standard): three questions are gates. If the label's meaning, the source owner or the permission is unknown or bad, the project is not approved as planned. Everything else can proceed with a dated action. Those three are gates because later engineering cannot repair them. Quality, freshness and change can be managed during the build; a model trained on the wrong label, or on data you were not allowed to use, has to be thrown away.
Count unknowns as well as bad answers. A project with no bad answers and seven unknowns is not low-risk. It is unexamined.
Running the intake session
Who attends. The sponsor; the named owner of each source (a person, not a team alias); someone who can speak for data protection; the engineering lead who will build it; and whoever owns quality. A source whose owner cannot be found is the first finding.
What to bring. One paragraph on the decision the AI will make or support, the candidate sources, and a sample extract from each. A few hundred records is enough to see the shape.
Agenda, 90 minutes. Ten minutes on the outcome. Sixty on the question set, source by source, writing each answer and grade into the intake record as you go. Twenty on gates and actions: what blocks approval, what needs evidence by a date, who owns each action.
Done when no gate question is unknown, every verbal answer has an evidence date, and the data owners have seen the record. Most spikes take days, such as profiling a sample or joining two systems to see what fraction of records link.
Tradeoffs and where this changes
Option | Gains | Costs | Best suited for | Warning signs |
Full intake before approval | Problems found while cheap | A week or two; owners' time | Production models, regulated or customer-facing data | The sponsor fills the form alone |
Gate questions only | Fast; catches what can't be repaired | Quality and change risks found later | Internal, low-risk tools | Scope grows with no second intake |
Intake after a prototype | Shows value before data work | Prototype sets expectations real data may not meet | Exploration on public or synthetic data | Prototype results quoted in the business case |
Two exceptions change the set itself. Retrieval assistants have no label, so that question falls away, but permission and sensitivity get harder because anything in the corpus can be shown to a user; see classifying data before you ingest it into RAG. Vendor models take the vendor's training data out of your hands, but the intake still applies to everything you send and index.
Implementation reality: where this breaks
The sponsor answers everything. The record looks complete, but every answer came from the person who wants approval. The fix is a rule: an answer about a source is documented only when its owner gave it.
Nobody owns the data. For shared drives, legacy databases and spreadsheets, often nobody will put their name to it. That is a finding, not an obstacle; who owns the data your AI uses covers assigning owners without a reorganisation.
The record goes stale. Re-run the gate questions when scope changes, when a source is added, and before go-live. The change questions tell you when the answers are about to expire.
For how intake fits with ownership, contracts, quality gates, lineage, classification and retention, see the anchor guide, data governance for AI delivery.
What to do next
Pick one AI project that is about to be approved, or already in flight. Answer the twelve questions yourself in an hour, grading each honestly, and count the gate questions you could not grade documented. Take that count, and the names of the people who could answer them, to the sponsor and ask for the ninety-minute session before the next commitment. Record the results in the AI Data Intake & Contract template, and where personal data is involved, check them against your AI Data Handling Policy.
Final takeaway
AI projects rarely fail because the model was the wrong choice. They fail because the plan rested on unchecked assumptions about data: that the label meant what its name said, that the data could be used, that the source would stay still. Each is a question with an answer, and the answer is cheapest at intake, asked of the people who own the data, while it can still change the scope instead of breaking the timeline.
Sources
Regulation (EU) 2024/1689 (AI Act), Article 10, Data and data governance (2024), with dates of application as amended (Annex III high-risk systems from 2 December 2027). https://eur-lex.europa.eu/eli/reg/2024/1689/oj
Regulation (EU) 2016/679 (GDPR), Article 5, Principles relating to processing of personal data (2016). https://eur-lex.europa.eu/eli/reg/2016/679/oj
Regulation (EU) 2016/679 (GDPR), Article 35, Data protection impact assessment (2016). https://eur-lex.europa.eu/eli/reg/2016/679/oj
NIST, AI RMF Playbook, MAP function, MAP 2.3 (companion to NIST AI 100-1, AI RMF 1.0, January 2023). https://airc.nist.gov/airmf-resources/playbook/map/
Frequently Asked Questions
When should data questions be asked in an AI project?
At intake, before the project is approved, staffed or dated. Data questions asked during a later "data preparation" phase arrive after the commitments they should have shaped, so a bad answer becomes a delay instead of a design change.
Who needs to be in an AI data intake session?
The business sponsor, the named owner of each data source the project would use, someone who can speak for privacy or data protection, and the engineering or product lead who will build it. If a source has no owner who can attend, that is the first finding.
What is the most important data question before an AI project?
For a supervised model, ask what the label actually records. Historical labels often record a past decision or a process step rather than the outcome you want to predict, and a model trained on them learns to repeat the decision.
