Why Your AI Is Probably Lying to You (And How to Catch It)

The Hospital, the Resume, and the $1,000 Promise

A hospital in Chicago spent eighteen months building a machine learning model that was supposed to predict which pneumonia patients were at high risk of complications. The model was elegant. It used deep neural networks, the cleanest data the hospital could find, and a team of eight PhDs. On paper, it looked brilliant. In tests, it achieved 92% accuracy. In a small pilot, it performed even better. The hospital got excited. So excited, in fact, that they briefly considered rolling it out across the entire system.

Then a doctor noticed something strange. The model was telling the team to send asthmatic pneumonia patients home. *Home.* Not to the ICU. Not to the monitored ward. Home, with a prescription and a follow-up appointment. This was, on its face, completely backwards. Asthmatic patients with pneumonia are among the sickest in any hospital. They have trouble breathing at baseline. Add an infection to their lungs and the situation can turn critical in hours. Anyone who has worked a night shift in an emergency room knows you never, ever send these patients home early.

But the model was confident. It was sure. It had been trained on thousands of previous cases and it knew, statistically, that asthmatic pneumonia patients in the training data often *did* go home without incident. The reason wasn't that they were healthy. The reason was that doctors, knowing these patients were high risk, admitted them aggressively, treated them early, and prevented complications. The model saw "low complications" and learned "low risk." It never learned that the doctors themselves were the safety net.

This is not a story about evil machines. It is not a story about AI gone rogue. It is a story about what happens when we build a system that gets a perfect score on the test, but the test was never measuring what we actually cared about.

---

About 2,000 miles east, inside a major tech company, a different machine was making a different kind of mistake. An AI hiring tool had been trained on a decade's worth of resumes submitted to the company. It learned, very efficiently, what successful candidates looked like. It learned the words they used, the schools they attended, the hobbies they listed, the formatting choices they made. Then it began screening new applicants.

The model worked. It was fast, consistent, and produced clear scores. It also quietly learned to prefer male candidates. Not because anyone told it to. Not because the team was sexist. It learned this from history. The historical resumes came from a company where, for ten years, more men than women were hired, more men were promoted, more men were kept around. The model picked up on subtle signals: certain verbs, certain extracurriculars, the names of certain women's colleges. It downgraded resumes that contained "women's" anywhere in them. Downgraded the word "women's" in "women's chess club chair." It was so effective at this hidden bias that the team only noticed when they started digging into why their pipeline had narrowed.

The model wasn't broken. It was working exactly as it had been trained to work. It just hadn't been trained on the right thing.

---

And then, in 2024, an airline's customer service chatbot made international headlines. A passenger had asked about the airline's bereavement policy after a relative died. The chatbot, in a moment of confident creativity, told the passenger that yes, the airline would offer a full refund under its bereavement policy. The passenger, grieving and relieved, made decisions based on that promise.

There was no such policy. The airline did not offer bereavement refunds. The chatbot had invented the policy. It had invented it because, when it looked at the patterns in its training data, generating a confident, helpful-sounding paragraph was the most likely next sequence of words. The airline eventually took the matter to court. The court ruled in the passenger's favor, noting that the company was responsible for what its AI said, even if the AI made it up.

A grieving customer. A confident machine. A policy that didn't exist. All because nobody had built a system to catch the difference between "sounds right" and "is right."

---

Three different industries. Three different problems. The same underlying pattern. These are not anomalies. They are the predictable output of a global industry that has, for the past decade, been racing to build smarter and smarter models while paying almost no attention to whether the models were actually measuring the right things.

Welcome to the evaluation crisis.

The Evaluation Crisis: How We Got Here

For most of the history of artificial intelligence, the field moved slowly. Progress was measured in decades, sometimes centuries. Then, sometime around 2012, something changed. A neural network called AlexNet crushed an image recognition competition. A few years later, a model called AlphaGo beat the world champion at Go, a game long thought to be uncrackable by machines. In 2020, GPT-3 demonstrated language abilities that startled even its creators. In 2022, ChatGPT reached 100 million users in two months, faster than any consumer technology in human history.

The pace was dizzying. And as the pace accelerated, a strange thing happened to how we measured progress.

The field developed a small number of standard tests. The most famous is probably the Massive Multitask Language Understanding benchmark, or MMLU, which tests models on 57 subjects from elementary math to professional law. There was HumanEval, which tested whether models could write code that passed specific tests. There was GLUE, SuperGLUE, BIG-bench, and dozens of others. Each year, new benchmarks appeared. Each year, models climbed the leaderboards.

And each year, the numbers got better. *Much* better. By 2024, the top language models were scoring in the 88th to 90th percentile on MMLU. They were passing bar exams. They were acing the SAT. They were outperforming most humans on most standardized tests of knowledge and reasoning. If you read the leaderboards, you'd be forgiven for thinking we were on the verge of building digital gods.

Then people started using these models in the real world. And the experience was, often, bewildering.

A model that scored 90% on a professional medical exam would confidently tell a patient to take a medication that would kill them. A coding assistant that passed 95% of HumanEval would generate code with subtle security vulnerabilities that a junior developer would have spotted. A customer service bot that ranked at the top of every benchmark would tell a customer to glue their cheese to pizza.

This is what I'm calling the evaluation crisis. It's the gap between *how well AI seems to be doing* and *how well AI is actually doing*. It's the chasm between benchmark performance and real-world performance. And it's getting wider, not narrower, with every new model release.

The reason is simple, even if the field doesn't like to admit it. Most AI systems are not being evaluated on the things that matter most. They are being evaluated on what is easy to measure, what is convenient to test, and what makes for a good press release. The result is a parade of models that ace their exams and fail their jobs.

The Benchmark Paradox

Let's make this concrete. Consider two language models. Model A and Model B.

Model A was trained by a well-resourced lab with vast compute, clean data, and a team of fifty. It scores 89% on MMLU, 92% on HumanEval, and 85% on a new reasoning benchmark called GSM8K. It produces fluent, confident, grammatical text. In conversations, it rarely contradicts itself. It is, by every standard benchmark, the better model.

Model B was trained by a smaller team with less compute. It scores 78% on MMLU, 80% on HumanEval, and 75% on GSM8K. Its text is occasionally awkward. It sometimes hedges or expresses uncertainty in ways that look unprofessional.