What Is AI Ethics, and Why Should You Care?

What Is AI Ethics, and Why Should You Care?

The Judge Who Wasn't Human

On a spring morning in 2013, a man named Bernard Loomis walked into a Wisconsin courthouse for a hearing that would change his life—and, eventually, change how we think about artificial intelligence in the justice system. Loomis had been convicted of stealing tools from a Home Depot, a relatively minor offense. But before his sentencing, something unusual happened. Instead of relying solely on a judge's gut feeling or a probation officer's recommendation, the court consulted a computer program called COMPAS. The system analyzed data about Loomis—his age, criminal history, the nature of his offense, even his zip code—and produced a score. That score predicted he was at "high risk" of reoffending.

Based in part on that number, the judge sentenced Loomis to six years in prison.

Loomis appealed, arguing that using a secret algorithm to influence his sentence violated his right to due process. He couldn't examine the program, challenge its methodology, or even understand exactly what factors it considered. The Wisconsin Supreme Court ultimately upheld the sentence, but the case ignited a national conversation that continues today. If a machine helps decide whether you go to prison, what should you know about that machine? Who makes sure it's fair? And what happens when it's wrong?

This isn't a story from a science fiction novel. It's happening in courtrooms across America right now, in 2023 and beyond. COMPAS, which stands for Correctional Offender Management Profiling for Alternative Sanctions, is just one of many algorithmic tools reshaping decisions about bail, sentencing, and parole. These systems promise objectivity—freedom from the biases that infect human judgment. But as we'll see throughout this book, they often carry their own biases, hidden in the data they learn from and the assumptions their creators make.

The Resume That Got Trashed for Being Female

Let's travel from the courthouse to the HR department. In 2014, Amazon was growing faster than any tech company in history, and its recruiting team faced a classic problem: too many job applications, not enough hours in the day to read them all. The solution seemed obvious—build an AI system that could screen resumes automatically, flagging the best candidates for human review.

Amazon's engineers got to work. They fed the system thousands of resumes from the previous decade, hoping it would learn to identify the qualities that made someone a good Amazon employee. The system would do the tedious first pass, and humans would make the final calls.

Within a year, the company realized the plan had a serious problem. The AI had learned to penalize resumes that included the word "women's"—as in "women's chess club captain"—and to downgrade graduates of all-women's colleges. In one telling incident, the system rescinded consideration for candidates who had the word "women" in their email address. The algorithm hadn't set out to discriminate against women. It had learned to do so by analyzing a decade of successful hires, most of whom were men—particularly men in technical roles.

Amazon scrapped the project in 2017, after WIRED reported on its existence. The company said the tool "was never used by Amazon HR to evaluate candidates." But the damage was done, and the story revealed something troubling about machine learning: systems can learn to discriminate not because anyone programmed them to, but because history is biased and data reflects that bias.

Think about what happened here. Amazon didn't want a discriminatory system. Their engineers didn't wake up one morning and decide to build a tool that would weed out women. The bias crept in silently, embedded in patterns that no one explicitly examined. The system was trained on past successes, and past successes reflected the gender composition of the tech industry in previous decades. When you train a machine learning system on biased data, you get a biased system.

This is one of the most important—and most counterintuitive—lessons of AI ethics: algorithms can be discriminatory without anyone intending discrimination. The bias doesn't come from a malevolent programmer; it comes from a malevolent history that the algorithm faithfully learned.

The Phone That Knows You're Pregnant

Now let's step away from courtrooms and corporations and enter something more intimate: your home, your body, your smartphone.

In 2012, a woman in Target, Minnesota received a catalog of baby products. Her father was furious—she was a teenager, and he assumed Target was trying to tempt her with marketing for infants. He complained to the store manager. A week later, the father called back, this time to apologize. His daughter was indeed pregnant; Target's algorithm had detected it before anyone in her family knew.

This story, which became famous when New York Times reporter Charles Duhigg wrote about it, illustrates both the power and the creepiness of modern algorithmic systems. Target's data scientists had discovered that pregnant women follow certain purchasing patterns: they buy vitamins, unscented lotions, cotton balls, and other seemingly unrelated items. By tracking these patterns across millions of transactions, the company's system could predict pregnancy with surprising accuracy—and send targeted advertisements before the expectant mothers had told anyone else.

The Minnesota father's reaction captures something important about how most of us feel about these systems. We appreciate convenience. We like it when our phone suggests a restaurant we actually want to try, or when our streaming service introduces us to a show we love. But there's something unsettling about a machine that knows us better than our families do, that sees patterns in our behavior we don't even recognize ourselves.

And Target's pregnancy predictions were just the beginning. Today's smartphones can track your sleep patterns, your heart rate, your steps, your location throughout the day. They know when you exercise, when you're stressed, when you're sick. Wearable devices like smartwatches can detect atrial fibrillation—a heart condition that increases stroke risk—before you feel any symptoms. Your phone's microphone might notice that your voice sounds different when you're developing a cold.

All this data is incredibly valuable, both to you and to others. Health insurance companies would love to know your risk factors. Pharmaceutical companies want to identify potential customers. Advertisers want to predict when you're vulnerable to certain messages. The question is: who controls this information? Who decides what happens to it? And what rights do you have when a machine draws conclusions about your body, your behavior, or your future?

These are the questions at the heart of AI ethics.

So What Exactly Is AI Ethics?

Let's step back for a moment and ask the obvious question: what do we mean when we talk about "AI ethics"?

The term can sound intimidating, like something only philosophers in tweed jackets debate in university lounges. But AI ethics is really just a practical discipline—it's about figuring out the rules and principles we need when machines start making decisions that used to require human judgment.

Consider the difference between a traditional computer program and an AI system. A traditional program follows explicit rules that programmers write: "IF the user enters a password, THEN check it against the database." You can trace every decision back to code that a human wrote.

An AI system, by contrast, learns from data. Instead of "IF age > 18 THEN allow access," you might have a neural network that looks at thousands of examples of people who were allowed or denied access and figures out the patterns on its own. This is incredibly powerful—these systems can find patterns humans would never notice—but it also means we often don't know exactly why an AI made a particular decision.