Foundations of Explainable AI for Deep Learning: Problem Definition, Taxonomies, and Evaluation Metrics

The tension between the unprecedented predictive performance of deep learning (DL) systems and their inherent opacity has emerged as one of the most critical barriers to their adoption in high-stakes domains such as healthcare, criminal justice, and autonomous systems. While state-of-the-art convolutional neural networks (CNNs) now achieve radiologist-level performance in detecting diabetic retinopathy from fundus photographs, and large language models (LLMs) outperform human experts on standardized medical licensing exams, the lack of transparency into their decision-making processes has led to regulatory rejections, clinician distrust, and ethical concerns around algorithmic bias. For instance, in 2021, the U.S. Food and Drug Administration (FDA) declined to approve a deep learning-based triage tool for emergency department patients until the developer could provide evidence of explainable decision logic that aligned with clinical standards, highlighting that predictive accuracy alone is insufficient for real-world deployment in regulated environments.^[1] This chapter formalizes the core problem of explainable AI (XAI) for deep learning, establishes a rigorous taxonomy of existing methods, outlines the unique challenges that distinguish DL explainability from traditional machine learning interpretability, and presents a mathematically grounded evaluation framework to assess the quality of explanations.

1.1 Formal Problem Definition of XAI for Deep Learning

At its core, the XAI for deep learning problem can be formalized as follows: given a trained deep learning model $M: \mathcal{X} \to \mathcal{Y}$, where $\mathcal{X}$ is a $d$-dimensional input space (e.g., $d=224 \times 224 \times 3 = 150,528$ for ImageNet-scale CNNs, $d=1024$ for text embeddings from 7B-parameter LLMs) and $\mathcal{Y}$ is an output space (e.g., discrete class labels for classification, continuous risk scores for regression, or bounding box coordinates for object detection), an explainer $E$ generates an explanation $e = E(M, x, y)$ for an input sample $x \in \mathcal{X}$ and its corresponding model prediction $y = M(x)$. Unlike traditional interpretable models (e.g., linear regression, decision trees) where the decision logic is directly observable from model parameters, deep learning models are *functionally opaque*: their decision logic is encoded in distributed, nonlinear interactions across millions to trillions of parameters, with no direct mapping between individual parameters and human-understandable concepts.

A valid explanation for deep learning must satisfy three mutually reinforcing objectives, which form the foundation of all XAI evaluation frameworks: 1. Faithfulness (Fidelity): The explanation must accurately reflect the model’s internal decision logic for the given input, rather than generating plausible but unrelated "hallucinations" that satisfy human expectations but misrepresent the model’s true reasoning. Formally, let $\phi(x, M, e) \in \mathbb{R}^d$ denote the feature importance vector generated by explainer $E$ for input $x$, model $M$, and explanation $e$. Faithfulness is quantified as the degree to which perturbations to features ranked as important by $\phi$ lead to consistent changes in the model’s prediction $y$, as formalized in Section 1.4. 2. Comprehensibility: The explanation must be interpretable by the target user group (e.g., clinicians, judges, software engineers) without requiring specialized expertise in deep learning. For example, an explanation stating "the model predicted malignancy because the tumor has spiculated margins" is comprehensible to a radiologist, while an explanation stating "the prediction is driven by a 0.82 activation in layer 17, channel 203 of the CNN" is not. 3. Utility: The explanation must enable the user to complete a specific downstream task, such as debugging model errors, auditing for algorithmic bias, or building trust in the model’s outputs. For example, an explanation that identifies a model’s reliance on hospital bed count rather than clinical symptoms to predict patient mortality is useful for auditing bias, while an explanation that only repeats the model’s prediction without new information is not.

A common misconception in XAI research is that the goal is to make deep learning models "fully transparent" by exposing all internal parameters and computations. For state-of-the-art models with trillions of parameters, this is computationally and cognitively infeasible. Instead, XAI aims to provide *sufficient explanation*: enough information about the model’s decision logic to meet the specific needs of the target user and use case. This distinction is critical for aligning XAI research with real-world deployment requirements, as the threshold for "sufficient" explanation varies dramatically across domains: a self-driving car’s perception model may only require explanations of why it detected a pedestrian to satisfy safety auditors, while a clinical diagnostic model may require explanations of how it weighed multiple clinical features to arrive at a diagnosis to satisfy regulatory requirements and clinician trust.

1.2 Rigorous Taxonomy of XAI Methods for Deep Learning

Existing XAI methods for deep learning can be organized along two orthogonal dimensions: (1) the time at which the explanation is generated relative to model training, and (2) the granularity of

Existing XAI methods for deep learning can be organized along two orthogonal, formally defined dimensions: (1) the temporal relationship between explanation generation and model training, and (2) the granularity of the explanation relative to the model’s decision logic. This taxonomy resolves longstanding ambiguities in XAI literature by distinguishing methods that are structurally embedded in model design from those applied externally to trained black-box models, and by clarifying the scope of explanations from individual predictions to global model behavior.

1.2.1 Dimension 1: Temporal Relationship (Intrinsic vs. Post-Hoc Interpretability)

The first dimension categorizes methods based on when explanations are generated relative to the model training process, a distinction first formalized by Lipton (2018) and later refined for deep learning systems by Murdoch et al. (2019).^[2]

• **Intrinsic interpretability** refers to models whose decision logic is directly observable from their structure and parameters at training time, without the need for external explanation tools. While early intrinsic interpretability research focused on traditional machine learning models (e.g., linear regression, decision trees, generalized additive models (GAMs)), recent advances in structured deep learning have extended intrinsic interpretability to deep systems. A canonical example is the Concept Bottleneck Model (CBM) proposed by Koh et al. (2020), which inserts a human-defined concept layer between the deep feature extractor and the prediction head. For a breast cancer pathology classification task, a CBM trained on whole-slide images outputs intermediate predictions for clinically validated concepts (e.g., "mitotic count > 10 per high-power field", "nuclear pleomorphism grade 3", "glandular structure disruption") before predicting malignancy. These intermediate concept predictions serve as direct, faithful explanations of the model’s decision logic, as they are explicitly optimized during training rather than inferred post-hoc. Intrinsic methods achieve near-perfect faithfulness by construction, as explanations are a direct byproduct of model training, but are limited by their expressive capacity: CBMs cannot capture concept interactions that are not predefined by domain experts, and GAMs cannot model high-order feature interactions without exponential increases in parameters.
• **Post-hoc interpretability** refers to methods applied to a trained black-box model after training is complete, to generate explanations for