Foundations of AI Evaluation: Taxonomy, History, and First Principles

Foundations of AI Evaluation: Taxonomy, History, and First Principles

1.1 The Evaluation Paradox

Modern artificial intelligence systems are subjected to a peculiar asymmetry: the components that determine their reliability, safety, and economic value—evaluation pipelines—are themselves the least engineered, least standardized, and least theoretically grounded artifacts in the entire machine learning lifecycle. A team may spend six months curating a pretraining corpus, another three months tuning a reward model, and another two months optimizing inference latency, yet allocate only a handful of days to the instrumentation that determines whether the resulting system is fit for release. This asymmetry is not a contingent oversight but a structural consequence of how the field has organized itself. Benchmarks emerged as ad-hoc community conveniences rather than as engineered instruments; leaderboards became the principal mechanism of scientific credit allocation; and the discipline of measurement, which in any mature empirical science precedes theory, was treated as a residual concern to be solved once a model "worked."

The consequence is a body of work in which claims of the form "model $X$ outperforms model $Y$ on task $Z$" are routinely advanced without specification of the inferential framework under which such a claim is licensed, the statistical power of the comparison, the operational construct the benchmark is intended to measure, or the population to which the resulting inference generalizes. The measurement crisis in AI is, in a precise sense, a crisis of construct validity: the field has built increasingly elaborate devices for producing numerical scores while remaining agnostic about what those scores denote. To take a single illustrative example, the MMLU benchmark has, at various points, been described in the literature as measuring "general knowledge," "multitask accuracy," "reasoning," "world knowledge," and "human-likeness"—five operationally distinct constructs whose differences have direct consequences for what an improvement of two percentage points actually means.

This chapter assumes that rigorous AI evaluation is a tractable engineering problem with a non-trivial conceptual prerequisite: a vocabulary of distinctions that any practitioner must internalize before instrumenting benchmarks, online feedback loops, red-teams, or business metrics. We call these distinctions *taxonomies* because they partition the space of possible evaluations along axes along which design choices and inferential interpretations differ. We further assume that no single taxonomy is sufficient; the entire apparatus of evaluation must be understood as a *portfolio* of complementary methods whose joint contribution exceeds the sum of any individual channel. The page that follows formalizes this portfolio, situates it in the historical lineage from which it descends, and provides a constructive artifact—the *evaluation charter*—that any serious AI initiative can instantiate.

1.2 A Historical Lineage of Evaluation Methodology

The genealogy of contemporary AI evaluation is shorter than commonly supposed and bears the topological scars of three distinct disciplinary ancestors: information retrieval, natural language generation, and psychometrics. Each contributed a metric family whose assumptions have quietly survived into the era of large language models, often without the awareness of the practitioners who now rely on them.

Phase I: The Information Retrieval Inheritance (1960s–1990s). The earliest formal evaluation frameworks in computer science arose in information retrieval, where the Cranfield experiments of the early 1960s established the template that subsequent generations would inherit: a fixed collection of documents, a fixed set of queries, and a set of relevance judgments supplied by human annotators against which system outputs could be scored. From this template emerged the now-canonical primitives of *precision* (the fraction of retrieved items that are relevant) and *recall* (the fraction of relevant items that are retrieved), along with their composite extensions, $F_{\beta}$ scores and mean average precision (mAP). These primitives encode a particular model of evaluation: the system produces a *set* of outputs, the world supplies ground truth as a *set* of relevant items, and quality is defined as overlap between these sets. The implicit assumptions—that relevance is binary, exhaustive, and stable across annotators—are non-trivial and rarely examined. When the same primitives were ported into retrieval-augmented generation, recommender systems, and even certain LLM evaluations, these assumptions travelled unannounced.

Phase II: The Natural Language Generation Heuristic Era (2000s–2010s). As machine learning began to be applied to language generation, the field needed metrics for outputs that are not sets but sequences, and not strictly right or wrong but variably adequate. The response was a generation of reference-based heuristics—BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), METEOR (Banerjee and Lavie, 2005), and chrF (Popović, 2015)—each of which computes a similarity score between a candidate output and one or more human-authored references. BLEU computes a geometric mean of modified $n$-gram precisions with a brevity penalty; ROUGE computes recall-oriented $n$-gram overlap. These metrics share a fundamental epistemic stance: they treat human-written references as a *proxy* for quality, and any deviation from the reference as a penalty. The stance has three consequences that practitioners often overlook. First, the metrics conflate semantic adequacy with lexical contiguity; a paraphrase that preserves meaning but shares few $n$-grams is penalized identically to a generation that has changed meaning. Second, the metrics are reference-dependent and therefore measure only what the reference set contains; an excellent output that happens to deviate from the references is rewarded less than a mediocre output that echoes them. Third, the metrics are insensitive to the *consequences* of generation—whether the produced text, if acted upon, would lead to correct downstream behavior. The translation and summarization communities built a generation of scientific literature on these proxies, and when the LLM revolution repurposed them as model-selection criteria, the proxies inherited their limitations.

Phase III: Holistic Multi-Metric Frameworks (2018–present). The recognition that single-number metrics fail to capture the multidimensional nature of model quality crystallized in a third phase. Two benchmark projects represent the watershed: HELM (Holistic Evaluation of Language Models; Liang et al., 2022) and BIG-Bench (Srivastava et al., 2022). HELM introduced a matrix-structured evaluation in which a fixed battery of seven metric categories—accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency—is measured across a wide range of tasks, with explicit attention to standardization of prompts and reporting of uncertainty. BIG-Bench contributed 204 diverse tasks contributed by community authors, deliberately spanning from trivially solvable puzzles to problems at the frontier of model capability. Both projects embodied a methodological commitment: that an evaluation result is not a number but a *vector* in a multidimensional space, and that any reduction to a scalar (e.g., an "average accuracy") is itself an editorial choice that requires justification.

This third phase also produced the most widely cited artifact of the era, Chatbot Arena (Zheng et al., 2023), which substituted human preference judgments in pairwise comparisons for ground-truth labels. Arena-style evaluation sits in productive tension with the HELM tradition: it gains ecological validity (real prompts, real users, blinded comparisons) at the cost of construct specificity (what, exactly, do users prefer when they prefer A to B?). The