The Interpretability Crisis in Deep Learning

The Interpretability Crisis in Deep Learning

A medical imaging model flags a malignant lesion. An autonomous vehicle fails to brake. A loan application is denied in eleven milliseconds. In each case, the deployed neural network reaches a high-confidence decision, yet neither the operator, the affected citizen, nor the engineer who built the system can articulate *why* the decision was made. This is the interpretability crisis: a widening gap between the predictive power of deep learning systems and our ability to interrogate, contest, and govern their behavior. For decades, the field treated accuracy and interpretability as opposing forces on a single axis—the more you want of one, the less you get of the other. Production reality, regulatory pressure, and a steady drumbeat of high-profile failures have made that framing untenable. Explainability is no longer a research curiosity reserved for academics. It is an engineering requirement, a legal obligation in many jurisdictions, and increasingly a precondition for deployment in any high-stakes domain.

This chapter establishes the formal and empirical foundation for the rest of the book. We begin by dismantling the accuracy–interpretability trade-off myth with empirical counterevidence from production systems. We then introduce a precise taxonomy that separates intrinsic interpretability from post-hoc explainability, local from global methods, and model-specific from model-agnostic techniques. The third section surveys the regulatory landscape—the EU AI Act, GDPR Article 22, sector-specific mandates in healthcare and finance, and the emerging US frameworks—arguing that explainability has become a binding legal requirement. In the fourth section we dissect three canonical failure cases: the COMPAS recidivism controversy, the unsafe recommendations of IBM Watson for Oncology, and the pervasive problem of spurious correlations in image classifiers. The fifth section translates intuition into mathematics, formalizing the explanation problem as a constrained optimization. We close with two engineering artifacts that will be referenced throughout the book: a reusable `Explainer` base class that anchors the practical chapters, and a deployment audit checklist for compliance and risk review.

1.1 The Myth of the Accuracy–Interpretability Trade-Off

The trade-off narrative is simple and seductive. Linear models are transparent: a positive coefficient on `income` means higher income increases the predicted probability. Decision trees can be printed on a single page. Generalized additive models (GAMs) decompose predictions into per-feature contributions a human can verify line by line. Deep neural networks, by contrast, are parameterized by hundreds of millions of weights arranged in deeply nested nonlinear compositions. A standard ImageNet ResNet-50 has $\sim 25.6$ million parameters; a contemporary large language model has hundreds of billions. The intuition that this expressivity must come at the cost of legibility has hardened into an article of faith. It is also empirically false more often than its proponents acknowledge.

The first systematic challenge came from Caruana and colleagues at Microsoft Research, who revisited a 1990s pneumonia risk prediction problem. The original researchers had used a rule-based system because clinicians required explanations. Caruana's team trained a gradient boosted tree, a random forest, a logistic regression, and a neural network on the same data. The neural network was barely competitive. The logistic regression and the boosted trees were both highly accurate *and* interpretable, with the boosted tree slightly outperforming logistic regression on AUC. A follow-up in 2015 with high-performance GAMs further improved the calibrated accuracy. The lesson was not that interpretable models were inadequate, but that *the field had stopped investing in them*. Modern tooling, better regularization, and large datasets had closed what appeared to be an interpretability gap.

Subsequent work has refined this finding. Rudin (2019) synthesized dozens of case studies and concluded that for many high-stakes tabular problems—credit scoring, criminal recidivism, hospital readmission—there exist *intrinsically interpretable* models whose accuracy matches the best black boxes within statistical noise. Where the trade-off does appear, it is concentrated in perceptual tasks: image classification, speech recognition, machine translation, and protein structure prediction. These are domains where the input is high-dimensional and the relevant features are not pre-specified. A doctor tells you which lab values matter; a photograph does not announce which pixels encode a tumor.

The trade-off narrative is also misleading along a second axis. Interpretability is not a single scalar that can be traded against accuracy. It has at least four orthogonal components: simulatability (can a human run the model mentally?), decomposability (can each input be understood?), algorithmic transparency (does the learning procedure admit theoretical guarantees?), and post-hoc explainability (can the trained model be interrogated?). A model can be strong on one dimension and weak on another. A deep network with attention is not simulatable, but it may be partially decomposable through attention heatmaps. A linear model is simulatable and decomposable but offers no algorithmic transparency about why the linear fit is the right functional form. Choosing between them is not a scalar trade-off but a multi-objective engineering problem with regulatory, ethical, and operational constraints folded in.

A third failure mode in the trade-off narrative is its conflation of *peak benchmark performance* with *deployment performance*. A deep model that achieves 99.2% top-1 accuracy on a curated test set may underperform a simpler model on the long tail of production traffic, on distribution shift, or on adversarial perturbations. Empirical studies of medical imaging models, including Zech et al. (2018) on pneumonia classification, have shown that state-of-the-art CNNs can exploit spurious correlations—hospital-specific metal markers in chest x-rays—that disappear the moment deployment context changes. A simpler model, less hungry for capacity, is more likely to learn the intended signal. Interpretability and robustness, in this regime, are complements, not substitutes.

None of this is an argument for abandoning deep learning. The opposite: deep learning remains the best tool we have for perceptual and generative problems. The argument is that the trade-off is conditional, domain-specific, and frequently exaggerated. In tabular and structured domains, we should default to interpretable models and reach for black boxes only with positive evidence. In perceptual domains, we should treat explainability as a separate engineering layer on top of the model, not as a property to be traded away. The rest of this chapter—and indeed the rest of this book—is built on that foundation.

1.2 A Formal Taxonomy of Interpretability and Explainability

Before discussing methods, we need vocabulary. The literature is notorious for using *interpretability*, *explainability*, and *transparency* interchangeably, which produces conversations in which two practitioners talk past each other. We adopt the following distinctions, which align with the consensus emerging from the ACM, IEEE, and major XAI survey papers.

Interpretability is a property of a model. A model is interpretable if a human can understand its entire decision-making process from its specification, parameters, and inputs. Linear regression is interpretable: the formula is the explanation. A small decision tree is interpretable: the path from root to leaf is the explanation. A logistic regression with engineered monotonic features is interpretable. A deep ResNet is not.

Explainability is a property of a method applied to a model. Given a model $f$ that may itself be