AI Bias: What It Is and Why It Happens
Updated: 5 hours ago
In 2019, a team led by Ziad Obermeyer published a study of an algorithm used by US health systems to decide which patients should get extra care management. It was typical of tools used across the industry and affected millions of patients. At any given risk score, Black patients were considerably sicker than white patients. Correcting the disparity would have raised the share of Black patients selected for extra help from 17.7% to 46.5%.
The algorithm didn't use race. The bias came from what it had been told to predict. It predicted future health care costs, as a stand-in for health need. Because of unequal access to care, less money was spent on Black patients who were just as sick, so the costs understated their need. The model learned the pattern accurately. The target was the problem.
That's the most useful thing to understand about AI bias, and the old version of this post didn't say it clearly. Bias usually enters before any model is trained: in what the data recorded and in what the system is asked to predict. Those are discovery decisions, and they can be checked.
What AI bias means
In this post, AI bias means a system's errors or outcomes are systematically worse for some groups of people than others. Not random mistakes, but patterned ones that repeat and can usually be found once you know where to look.
NIST's 2022 report on identifying and managing bias in AI groups the sources into three categories:
Systemic bias: from institutions and society, already present in how decisions were made and recorded before any model existed.
Statistical and computational bias: from the data and methods, such as samples that don't represent everyone or measurements that work better for some groups.
Human bias: from the people designing, building and using the system, including how they interpret its outputs.
The NIST report makes the point that technical fixes, like more representative datasets, address only part of this. Systemic and human sources are significant and often overlooked. The cases below show each.
The test: for any model that makes or informs decisions about people, write down what it predicts, where the training labels came from, and who decided both. If nobody can answer, the most likely source of bias hasn't been examined.
Three cases, checked against their sources
A hiring model trained on past hiring. In 2018, Reuters reported that Amazon had abandoned an experimental recruiting tool by the start of 2017. The models had learned from resumes submitted over a ten-year period, most of them from men, and the system taught itself that male candidates were preferable: it penalised resumes containing the word "women's". Amazon said the tool "was never used by Amazon recruiters to evaluate candidates"; the people Reuters spoke to said recruiters looked at its recommendations but never relied on them alone. This is systemic bias in its simplest form: past decisions became the definition of a good candidate.
Facial analysis that works worse for some faces. Joy Buolamwini and Timnit Gebru's 2018 Gender Shades study evaluated three commercial gender classification systems on a dataset balanced by gender and skin type. Darker-skinned women were the most misclassified group, with error rates of up to 34.7%. The maximum error rate for lighter-skinned men was 0.8%. This is statistical bias: a system that performs well on average can fail badly for a group under-represented in its data. The study's method matters as much as its numbers: it reported accuracy for each group, not just overall.
Lending, where algorithms reduced discrimination but didn't remove it. A study of US mortgages by Robert Bartlett and colleagues, published in 2022, found that lenders charged otherwise-equivalent Latinx and African-American borrowers 7.9 basis points more for purchase mortgages and 3.6 more for refinances, costing about $765 million a year in extra interest. The old version of this post attributed that figure to algorithms. The study says otherwise: the $765 million is across lenders, and algorithmic lenders discriminated about 40% less than face-to-face lenders, with no discrimination in rejection rates. They didn't eliminate it, possibly because algorithms charged more where borrowers shopped around less. The useful lesson is narrower than "algorithms discriminate": automating a decision can reduce bias and still leave some, and you only know which by measuring.
The test: for each case, name the point where the bias entered: the labels, the sample, or the pricing logic. It's never "the AI" in general, and the fix depends on the specific point.
Why it happens: four mechanisms
The labels record past decisions
A model learns to predict its labels. If the labels are past human decisions (who was hired, who was referred for investigation, who was approved), the model learns those decisions, including their biases. The Amazon tool is the standard example.
The target is a convenient proxy
Often the thing you care about can't be measured directly, so the model predicts something that can. The Obermeyer study's authors warn that "the choice of convenient, seemingly effective proxies for ground truth can be an important source of algorithmic bias." Cost for need, arrests for crime, clicks for interest: each proxy is accurate about itself and biased as a measure of the real thing.
The sample under-represents some people
If some groups are rare in the training or test data, the model's errors for them are both larger and less visible. An overall accuracy figure hides them, which is exactly what Gender Shades exposed.
Features stand in for protected characteristics
Removing race, sex or age from the inputs doesn't remove their influence if other features carry the same information: postcode, name, school, employment gaps, even how a form was filled in. The model doesn't need the protected attribute if a proxy predicts it.
The test: list the model's top features by importance and ask, for each, whether it could be standing in for a protected characteristic. Then ask the same of the label.
A worked example: one model, two checks
(Composite.) An insurer uses a model to flag claims for review by its special investigations unit. A flagged claim is delayed while investigators look at it. The model was trained on five years of claims, with the label "was referred to investigators". Before a planned expansion, the quality lead runs two checks on a month of claims, split by age band because age is a protected characteristic in the insurer's market.
Check 1: outcome rates by group. What share of each group is flagged? The US federal guidelines on employee selection give a widely used rule of thumb, the four-fifths rule: a selection rate for any group that is less than four-fifths (80%) of the rate for the most favoured group is generally regarded as evidence of adverse impact. The guidelines also say smaller differences can still count when they're significant in statistical and practical terms. The rule comes from employment law, but the calculation is a useful first screen for any decision system.
Check 2: error rates by group. Of the claims flagged in each group, what share turned out to be fraud?
Measure (one month) | Under 65 | 65 and over |
Claims | 18,000 | 4,000 |
Flagged for investigation | 6.0% | 10.5% |
Not flagged (the favourable outcome) | 94.0% | 89.5% |
Ratio of favourable rates | 89.5 ÷ 94.0 = 0.95 | |
Flagged claims confirmed as fraud | 28% | 12% |
The first check passes easily: 0.95 is well above 0.8. The second shows the problem. Older claimants are flagged nearly twice as often, and their flags are less than half as likely to be right. Most of the extra flags are false alarms, and each one delays a genuine claim from an older customer.
The cause was in the label. For years, claims submitted on paper forms had been routinely referred to investigators, and older customers were more likely to use paper. The model learned "paper form" as a fraud signal, and paper stood in for age. The team retrained on confirmed fraud outcomes instead of referrals, removed the form-type feature, and reran both checks before the expansion.
The example shows why one check isn't enough. A rate-based test can pass while the errors fall unevenly, and an error-based test needs reliable outcome data, which you may only have for the cases that were investigated. Fairness metrics: what they measure and where they conflict covers why different fairness measures can't all be satisfied at once, and how to choose.
The test: for a model you run, compute both checks for each group you can lawfully and reliably identify: the favourable-outcome rate and its ratio to the best-treated group, and the error rate. If you can't compute the second, find out why: missing outcome data is itself a finding.
The "AI is objective" myth
The most dangerous belief about AI bias is that a computer's decision is neutral because a computer made it. Each mechanism above is a human choice: which decisions became labels, which proxy became the target, which data was collected, which features were allowed. Calling the output objective doesn't remove those choices. It makes them harder to see and harder to challenge, because the person affected can't ask the model why.
The lending study is a useful corrective in the other direction too. Algorithms aren't automatically worse than people; in that market they were less discriminatory than face-to-face lending. The question is never "AI or humans?" It's "how does this system's outcome and error compare across groups, and compared with the process it replaces?"
What reduces it
Each mechanism has a matching control, and most of them are discovery work done before or alongside modelling.
Mechanism | Control | When |
Labels record past decisions | Audit the label: whose decision was it, and was it fair? Prefer outcomes over decisions | Before training |
Target is a proxy | Write down what the target measures and what it stands in for; check the gap by group | Before training |
Sample under-represents groups | Report accuracy and error per group, not only overall; collect more data where it's thin | Training and testing |
Features act as proxies | Test whether features predict protected characteristics; remove or constrain the strong ones | Training |
All of them, over time | Rerun both checks on live decisions on a schedule; monitor for drift | In production |
Two things sit on top. Human oversight of high-stakes decisions (hiring, lending, health, insurance, justice) creates a point of accountability, though it only helps if the reviewer can see the evidence and is free to disagree. Transparency means people affected should know a model was involved and be able to challenge the result.
Regulation is starting to require these checks. New York City's Local Law 144, enforced since July 2023, bars employers from using an automated employment decision tool unless it has had a bias audit within the past year, with a public summary of the results and notice to candidates. The EU AI Act treats many systems used in employment, credit and access to essential services as high-risk, with obligations to match; the classification guide covers which systems qualify.
The test: take one model and fill in the table above with what your team actually does today. Any blank row is the place to start, and the label audit is usually the cheapest and most revealing. Bias in training data: where it hides and how to find it goes deeper on that step.
What to do this week
Pick one model or automated rule that makes decisions about people. Write down what it predicts and where its labels came from. Then compute the two checks for the groups you can identify: the favourable-outcome rate (with its ratio to the best-treated group) and the error rate. If either looks uneven, trace it back through the four mechanisms before touching the model itself.
Final takeaway
AI bias is rarely a mysterious property of the model. It usually enters through the labels, the target, the sample or a proxy feature, and every one of those is a human choice made before or during training. Ask what the system predicts and whose decisions it learned from, and check both outcome rates and error rates for each group, because either can look fine while the other doesn't. Then compare the system with the process it replaced: the goal is a fairer decision, which you can only show by measuring.
Next in the AI Ethics for Everyone learning path: We'll look at how AI bias actually gets measured — the metrics, the methods, and what "fair" even means when different definitions of fairness can contradict each other.
To build these checks into your release process, the AI Governance in the Pipeline checklist puts subgroup results in the model card, fairness bounds in the release gates and fairness drift in production monitoring.
Sources
Obermeyer, Z., Powers, B., Vogeli, C. and Mullainathan, S., "Dissecting racial bias in an algorithm used to manage the health of populations", Science 366(6464), 447–453 (2019): at a given risk score Black patients were considerably sicker; remedying the disparity would raise the share receiving extra help from 17.7% to 46.5%; the algorithm predicted cost rather than illness. https://doi.org/10.1126/science.aax2342
Schwartz, R. et al., Towards a Standard for Identifying and Managing Bias in Artificial Intelligence, NIST Special Publication 1270 (March 2022): three categories of bias in AI (systemic, statistical and human). https://doi.org/10.6028/NIST.SP.1270
Dastin, J., "Amazon scraps secret AI recruiting tool that showed bias against women" (Reuters, 10 October 2018). https://www.reuters.com/article/us-amazon-com-jobs-automation-insight-idUSKCN1MK08G
Buolamwini, J. and Gebru, T., "Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification", Proceedings of Machine Learning Research 81 (2018): three commercial systems; error rates up to 34.7% for darker-skinned females; maximum 0.8% for lighter-skinned males. https://proceedings.mlr.press/v81/buolamwini18a.html
Bartlett, R., Morse, A., Stanton, R. and Wallace, N., "Consumer-lending discrimination in the FinTech era", Journal of Financial Economics 143(1), 30–56 (2022): 7.9 (3.6) bps higher rates for purchase (refinance) mortgages, $765M yearly; algorithmic lenders reduce rate disparities by more than a third and show no discrimination in rejection rates. https://doi.org/10.1016/j.jfineco.2021.05.047
Uniform Guidelines on Employee Selection Procedures, 29 CFR § 1607.4(D): the four-fifths rule and its caveats. https://www.law.cornell.edu/cfr/text/29/1607.4
NYC Department of Consumer and Worker Protection, "Automated Employment Decision Tools (AEDT)": Local Law 144 bias audits, enforcement from 5 July 2023. https://www.nyc.gov/site/dca/about/automated-employment-decision-tools.page
Related reading
Keep learning. This article is part of the AI Governance & Responsible AI path in the ShiftQuality Learning Center. Ship AI you can stand behind — fair, transparent, and accountable.
