Why AI Evaluation Matters: The Stakes, The Challenges, The Opportunity

Why AI Evaluation Matters: The Stakes, The Challenges, The Opportunity

In 2018, an autonomous vehicle operated by Uber struck and killed Elaine Herzberg as she crossed a road in Tempe, Arizona. The safety driver was looking away from the road. The vehicle's sensors detected the pedestrian but the system's software had been tuned in a way that classified her as a false positive—another vehicle might have braked unnecessarily for a plastic bag blowing across the road, and the evaluation thresholds had been adjusted to reduce such "nuisance" braking. The algorithm was optimized for the wrong thing. One year earlier, a COMPAS recidivism algorithm used by judges in sentencing decisions was found to falsely flag Black defendants as higher-risk at nearly twice the rate of white defendants. In 2016, Microsoft's Tay chatbot launched on Twitter and began posting inflammatory content within 16 hours, forcing an emergency shutdown. These are not edge cases or anomalies—they are symptoms of a systematic failure to evaluate AI systems rigorously before deployment.

The question this chapter addresses is deceptively simple: Why should organizations invest in systematic AI evaluation, and what are the risks of neglecting it? The answer requires understanding not just the technical dimensions of AI quality, but the organizational, economic, and human factors that determine whether an AI system succeeds or fails in practice. We will examine documented failures that cost lives, careers, and billions in market value. We will explore the unique challenges that make AI evaluation fundamentally different from traditional software testing. We will introduce a framework—the four pillars of AI evaluation—that structures the approach taken throughout this book. And we will give you the tools to assess where your organization stands today and where to focus improvement efforts.

The stakes have never been higher. As AI systems take on increasingly consequential decisions—from diagnosing cancer to approving loans to controlling vehicles—the cost of getting evaluation wrong escalates dramatically. Yet despite this reality, most organizations continue to underinvest in the assessment layer of their AI development processes. This chapter establishes why that pattern must change.

AI Failures Have Real Consequences: Case Studies of Evaluation Gaps Causing Harm

The failures that have made headlines represent only the most visible portion of a much larger pattern. To understand why evaluation matters, we must first understand what happens when it is absent, inadequate, or misdirected.

The Uber ATG Incident: Optimizing for the Wrong Metric

The March 2018 fatality involving an Uber Advanced Technologies Group self-driving vehicle did not occur because the technology failed outright. The LiDAR sensors detected an object. The cameras recorded the scene. The problem was in how the system evaluated and prioritized that information.

The National Transportation Safety Board investigation revealed that Uber's self-driving system had a known issue with "phantom braking"—the vehicle would sometimes brake unnecessarily for stationary objects, plastic bags, or other non-hazards. This created discomfort for the safety drivers and potentially confusing situations for other road users. Uber's response was to adjust the detection and braking system to be less sensitive to obstacles. The threshold for emergency braking was raised. Objects that would previously have triggered immediate stopping were now classified as requiring further analysis.

The system that should have protected Elaine Herzberg had been deliberately tuned to underweight certain types of detection. This was not a hidden bug or an unforeseeable failure. It was the predictable result of optimizing for passenger comfort metrics while failing to evaluate the system's performance against safety requirements. The evaluation gap was not that no one was testing—the vehicle had been tested extensively on test tracks and roads. The gap was that the evaluation framework itself was misaligned. Success was measured in miles driven without disengagement, not in the accuracy of hazard detection across all scenarios.

The lesson is not simply "test more." It is that evaluation must ask the right questions. Testing the wrong thing thoroughly still produces a system that fails when it matters most.

Amazon's Automated Recruiting Tool: Biased Training Data

Amazon began developing an automated candidate screening tool in 2014. The system was trained on resumes submitted over a ten-year period—a dataset dominated by successful candidates from a period when the tech industry was predominantly male. The AI learned to associate male-dominated signals with successful hires. It downgraded resumes that included the word "women's" (as in "women's chess club") and penalized graduates of all-women's colleges.

The system was never deployed for formal hiring decisions, but it was used experimentally for several years before being abandoned in 2017. By the time Amazon scrapped the project, it had already influenced recruiting decisions and reinforced patterns of exclusion in a company that was trying to improve its diversity record.

The evaluation gap here was not in testing accuracy. The system was, in a narrow sense, accurate—it reliably predicted which candidates would advance given historical patterns. The gap was in recognizing that the training data encoded historical discrimination, and that the evaluation framework needed to include fairness and bias metrics alongside performance metrics. An adequate evaluation would have revealed the systematic disadvantage being baked into the model before it touched a single real candidate.

This case illustrates a crucial point: evaluation is not merely a technical exercise conducted at the end of development. It is a continuous process of asking whether the right thing is being built, not just whether the thing being built is right.

IBM Watson for Oncology: Mismatched Expectations and Inadequate Validation

IBM Watson for Oncology was launched with significant fanfare as a tool that would revolutionize cancer treatment. By analyzing vast amounts of medical literature and patient records, the system would provide treatment recommendations to oncologists. The commercial launch in 2013 was followed by rapid global deployment, with the system adopted by hospitals in the United States, Asia, and Europe.

By the late 2010s, reports emerged of Watson providing unsafe and incorrect treatment recommendations. Internal documents showed that the system had been trained on a relatively small number of synthetic cases rather than real patient data, and that oncologists at partner institutions had identified numerous instances where the system's recommendations conflicted with established clinical guidelines. In 2018, STAT News obtained internal IBM documents describing numerous instances of Watson recommending "unsafe and incorrect" treatments, including recommending drugs for cancer types where those drugs were contraindicated.

The evaluation gap in this case was multidimensional. The system had not been adequately validated against real-world patient outcomes. The training data did not reflect the complexity of actual clinical practice. The evaluation protocols did not include rigorous comparison against oncologist consensus. And perhaps most critically, the business pressures to launch quickly outpaced the scientific rigor required to validate such a consequential system.

The Pattern Across Cases

These are not isolated incidents. A 2019 study by the AI Now Institute identified recurring patterns in AI failures across multiple sectors: healthcare algorithms that performed worse for Black patients, hiring tools that discriminated against women, predictive policing systems that reinforced existing biases, and financial algorithms that denied loans to qualified minority applicants at disproportionat