AI Transparency: Explaining Decisions to the People They Affect

The previous posts in this path covered responsible AI beyond the checkbox, fairness metrics, and bias in training data. This post covers the commitment that ties them together: transparency — the practice of making AI systems understandable to the people they affect.

When an AI system denies a loan application, flags a resume for rejection, sets an insurance premium, or recommends a medical treatment, the person on the receiving end of that decision has a reasonable question: why? "The model scored you below the threshold" is not an explanation. "Your debt-to-income ratio of 45% exceeded the guideline of 40%, and your credit history showed three missed payments in the last year" is an explanation. The difference is accountability.

A system that cannot explain its decisions cannot be held accountable for them. And a system that cannot be held accountable should not be making decisions that affect people's lives.

The Spectrum of Interpretability

AI models exist on a spectrum from fully interpretable to effectively opaque.

Inherently interpretable models — linear regression, decision trees, logistic regression, rule-based systems — produce decisions that can be directly traced to input features. A linear regression model for loan approval has coefficients: each coefficient shows exactly how much each feature (income, debt ratio, credit score) contributes to the decision. The explanation is built into the model.

Post-hoc explainability applies to complex models (neural networks, ensemble methods, large language models) that are not directly interpretable. Techniques like SHAP, LIME, and attention visualization provide after-the-fact explanations of why the model made a specific prediction. These explanations are approximations — they tell you what the explainability method thinks influenced the decision, which may not perfectly represent the model's actual reasoning.

Effectively opaque models — large neural networks, deep ensembles, large language models — make predictions through billions of parameters in ways that resist simple explanation. Post-hoc techniques provide partial insight, but the full causal chain from input to output is not human-comprehensible.

The design decision: match the interpretability requirement to the stakes of the decision. For high-stakes decisions (credit, employment, healthcare, criminal justice), interpretable models or robust post-hoc explanations are necessary. For low-stakes decisions (product recommendations, content ranking), less interpretability may be acceptable — though transparency about the system's existence and general behavior is still warranted.

Explanations for Different Audiences

The same decision requires different explanations for different people.

The affected person needs a plain-language explanation of what factors influenced the decision and what, if anything, they can do differently. "Your application was declined because your annual income of $45,000 is below the $50,000 threshold for this loan product. You may qualify for our alternative product, or you could reapply if your income changes." This explanation is actionable and respectful.

The regulatory body needs a technical explanation of how the model works, what data it uses, how it was validated, and how fairness was assessed. This audience needs documentation, not a conversation — model cards, validation reports, fairness assessments, and audit trails.

The development team needs debugging-level explanations — feature importances, prediction confidence, similar cases with different outcomes, and edge cases where the model behaves unexpectedly. This audience needs tools, not documents — interactive dashboards, explanation APIs, and comparison interfaces.

Designing for all three audiences means building explanation capabilities at multiple levels of abstraction, from technical feature attributions to human-readable decision summaries.

Post-Hoc Explainability Techniques

When the model itself is not interpretable, post-hoc techniques provide explanations.

SHAP (SHapley Additive exPlanations) assigns each feature a contribution value for a specific prediction. For a loan denial, SHAP might show: credit score contributed -0.3 (toward denial), income contributed +0.1 (toward approval), debt-to-income ratio contributed -0.4 (toward denial). The contributions sum to the model's prediction. SHAP has strong theoretical foundations in game theory and provides consistent, locally accurate explanations.

LIME (Local Interpretable Model-agnostic Explanations) fits a simple interpretable model (like a linear model) to the neighborhood around a specific prediction. The interpretable model approximates the complex model's behavior for inputs similar to the one being explained. LIME then explains the simple model, which is a proxy for the complex model's behavior in that region.

Counterfactual explanations answer the question "what would need to be different for the outcome to change?" Instead of explaining why the loan was denied, a counterfactual explanation says "the loan would have been approved if your debt-to-income ratio were 38% instead of 45%." Counterfactual explanations are naturally actionable — they tell the person what to change.

Each technique has limitations. SHAP can be computationally expensive for large models. LIME explanations can be unstable — similar inputs can produce different explanations. Counterfactual explanations can suggest changes that are unrealistic or unethical (you cannot suggest someone change their age or zip code). Use multiple techniques and compare their outputs for high-stakes decisions.

Building Explanation Systems

Explainability is not a one-time analysis. It is a system that produces explanations at scale, in real time, as the model makes decisions.

The architecture: the model makes a prediction. The explanation service takes the prediction, the input features, and the model, and generates an explanation. The explanation is stored alongside the prediction for auditing. A human-readable version is presented to the affected person or the decision-making team.

For real-time decisions (loan applications, hiring screens), the explanation must be generated within the same response time as the prediction. Pre-computed SHAP values or lightweight LIME approximations enable this. For batch decisions (insurance pricing, credit limit adjustments), explanations can be computed offline and attached to the decision record.

The explanation should be tested like any other system output. Does the explanation accurately reflect the model's behavior? Do explanations for similar inputs produce similar explanations (consistency)? Do the explanations make sense to the intended audience (usability)? Do the explanations satisfy regulatory requirements (compliance)?

The Right to Explanation

The EU's General Data Protection Regulation (GDPR) includes provisions for explanation of automated decisions. The exact scope is debated by legal scholars, but the direction is clear: people affected by automated decisions have a right to meaningful information about the logic involved.

Beyond legal requirements, the ethical argument is straightforward. If your system makes a decision that affects someone's access to credit, employment, housing, insurance, or healthcare, that person deserves to understand why. Not because the law requires it — though it may — but because treating people as subjects of opaque algorithmic decisions rather than informed participants is a failure of respect.

The practical argument is also strong. Explainability catches model errors. When an explanation does not make sense — "the loan was denied primarily because the applicant's first name starts with J" — it reveals a model that has learned a spurious correlation. Explainability is a debugging tool as much as a transparency tool.

The Takeaway

AI transparency is the practice of making automated decisions understandable to the people they affect. Interpretable models provide built-in transparency. Post-hoc techniques (SHAP, LIME, counterfactual explanations) provide transparency for complex models. Explanation systems produce explanations at scale.

The standard is not "we can explain it to a data scientist." The standard is "the person affected by this decision can understand why it was made and what, if anything, they can do about it." Meeting that standard requires designing for explanation from the beginning — not adding it as an afterthought when a regulator asks.

---

Next in the "Responsible AI Practice" learning path: We'll cover AI impact assessments — how to evaluate the societal effects of an AI system before it is deployed, not after harm has occurred.