Foundations of AI Evaluation: Epistemology, Methodological Pluralism, and the Crisis of Measurement

1.1 The Three-Layer Ontology of AI Quality

When a vendor claims that its model "scores 92 on MMLU," or that a new conversational system "wins 68% of head-to-head comparisons against the previous generation," what epistemic warrant does that number carry? The uncomfortable answer, which motivates everything that follows, is that the warrant is fragile, conditional, and routinely over-extended by the very practitioners who report the numbers. To build any defensible evaluation system, we must first establish a precise ontology of what is being measured. The most common source of evaluative error in the field is the failure to keep distinct the three layers at which an AI system can possess "quality."

Capability denotes the latent competence of a system — the distribution of tasks it could in principle perform, conditional on some appropriate elicitation regime, prompting scaffold, tool access, and resource budget. Capability is a counterfactual property: it answers the question, "What could this system do under the right conditions?" A model that has internalized the rules of chess possesses the capability to play chess, even if no chess prompt has yet been presented. Capability is the construct most directly analogous to a person's potential or competence in cognitive science, as distinct from performance. The standard mechanism for probing capability is the *capability elicitation* protocol: an investigator constructs prompts, chain-of-thought scaffolds, or tool-use interfaces specifically designed to remove extraneous obstacles so that any latent capacity, if present, can be expressed. The Beyond the Imitation Game Benchmark (BIG-bench), with its 204 tasks contributed by hundreds of human authors, was conceived explicitly as a capability probe rather than a performance benchmark; many of its subtasks were constructed so that even a single successful solution would be informative about a system's latent repertoire.

Behavior denotes the observed performance of a system under a specific, fixed regime of inputs, prompts, decoding parameters, and operational constraints. Behavior is what the system actually does, not what it could do. The gap between capability and behavior can be vast, and it is the most common locus of the failures that recur throughout the rest of this book. A system may have the capability to solve a math problem yet exhibit the behavior of refusing, hallucinating an intermediate step, or being derailed by an irrelevant detail in the prompt. Behavior is the natural target of benchmark evaluation, because benchmarks are operationalizations of behavior: they fix the input distribution, the decoding settings, and the scoring function, and they observe the outputs. A score on MMLU is a behavior measurement under a specific protocol; the claim that it implies broader capability is an inference, not a measurement.

Impact denotes the downstream consequences of the system's behavior in a real or simulated environment. Impact is the layer at which capability and behavior are cashed out for users, organizations, and society. Did the recommendation system increase click-through? Did the code assistant reduce post-release bug counts? Did the medical triage tool improve patient outcomes? Did the chatbot generate reputational damage? These are questions about impact, and they are notoriously difficult to attribute because they require counterfactual reasoning, confounding control, and long causal chains. The standard mechanisms for probing impact are randomized controlled trials (A/B tests), quasi-experimental designs (difference-in-differences, synthetic controls, instrumental variables), and field studies. Impact is also the layer at which value judgments become most inescapable: a system that increases revenue but reduces customer trust has generated a different impact than one that does the reverse, and the evaluator must decide which dimension to privilege and how to weight the trade-off.

```mermaid flowchart TD A[Capability: Latent Competence] --> B[Behavior: Observed Performance] B --> C[Impact: Downstream Consequences] A1[Capability Probes: BIG-bench, knowledge batteries, skill elicitation] -.->|measure| A B1[Behavioral Probes: MMLU, HELM, task suites, unit tests] -.->|measure| B C1[Impact Probes: A/B tests, field studies, business KPIs, audits] -.->|measure| C ```

The diagram above makes the ontology explicit. Each layer is probed by a different family of evaluation modalities, and the arrows between layers indicate inferential leaps that evaluation systems routinely make but rarely justify. When a benchmark report claims that a high MMLU score "implies" better downstream performance in customer support, it is asserting an inference from behavior to impact without supplying the construct-validity evidence that would make that inference defensible. The cost of these unjustified inferences, aggregated over thousands of model releases and procurement decisions, is the central concern of this book. Subsequent sections of this chapter, and many of the chapters that follow, can be read as successive elaborations of the question: under what conditions is each of these inferential leaps legitimate, and how can an evaluation system be architected to make the leaps auditable rather than implicit?

A further subtlety deserves emphasis: the three layers are not strictly ordered in difficulty or importance, and they trade off against one another in ways that have structured the entire history of the field. Capability evaluation is intellectually appealing because it attempts to characterize the system in itself, abstracted from any particular deployment; but it is often criticized for being unmoored from practical utility. Impact evaluation is practically indispensable because it speaks to the only question that ultimately matters to most stakeholders; but it is so confounded, expensive, and slow that it cannot be the primary mode of evaluation during iterative model development. Behavior evaluation — benchmarking — sits in the middle and has therefore become the de facto workhorse of the field. The consequence, however, is that the field has over-invested in behavior measurement and under-invested in the validation of the inferential bridges between layers. A central argument of this book is that an architecturally sound evaluation system must explicitly allocate measurement effort across all three layers, with the allocation determined by the decision the evaluation is meant to inform. The four-pillar framework previewed in Section 1.8 is the operationalization of that allocation.

1.2 Construct Validity, Reliability, and Ecological Validity

The three-layer ontology tells us *what* we are trying to measure. The next conceptual move is to import a measurement framework that tells us *how well* we are measuring it. Psychometricians confronted a problem isomorphic to the one facing AI evaluators long before the term "AI evaluation" existed: how do you defend the claim that a numerical score on a test reflects a real underlying attribute of the person being tested? The framework they developed — construct validity — is directly applicable, and its explicit importation into AI evaluation is overdue.

Construct validity, in the canonical Cronbach and Meehl formulation, is not a single property of a measure but an ongoing program of evidence accumulation. A measure has construct validity to the extent that its scores correlate with other measures of the same construct (convergent evidence), discriminate between groups known to differ on the construct (discriminant evidence), and respond to experimental manipulations that theory predicts should affect the construct (manipulation evidence). Reliability — the consistency of measurement under repeated application — is a necessary but not sufficient condition for validity: a broken clock is perfectly reliable but completely invalid. Ecological validity — the extent