The AI That Rejected Maria: When Code Meets Fairness

> "The neutral man is a myth. Every tool carries the ideology of its creator." > — Langdon Winner, *The Whale and the Reactor*

The wind off the Delaware River cut through Maria’s thin coat as she hunched over her phone on the North Philadelphia bus stop bench, her 4-year-old daughter’s small hand curled tight around her sleeve. The rejection email from the city’s Office of Data and Innovation sat open on her screen, the words blurring through the tears pricking at her eyes: *“Thank you for your interest in the Junior Data Analyst role. After careful review, we have decided to move forward with other candidates whose qualifications more closely align with our needs.”*

She had applied for the job three times in the past year. Each time, the same form rejection. Each time, she assumed a human hiring manager had skimmed her resume and moved on: 5 years coordinating community services at a local nonprofit, a self-taught command of SQL and Python, a Coursera certificate in data analytics, and a portfolio project where she’d used public data to cut food pantry wait times by 32% in her neighborhood. None of it seemed to matter.

It took a friend who worked as a paralegal at a local civil rights firm 3 months to get to the bottom of the rejection. The city used an AI hiring tool called HireSense to screen 92% of its entry-level job applications, and Maria’s resume had scored a 42 out of 100 on the tool’s proprietary ranking system—12 points below the 54-point threshold for even a human review. When the legal team dug into the tool’s training data, they found the pattern: over the prior decade, every junior analyst hired by the city had been a white man with a 4-year degree from a selective private university, living in the city’s wealthy western suburbs. The AI had learned to associate Maria’s features—her state university education, her North Philadelphia zip code, the line on her resume about “coordinating parent engagement programs” that hinted at caregiving responsibilities—with low job performance, even though there was no evidence to support that link.

Maria’s story is not an outlier. It is the new normal for millions of people navigating a world where algorithms make life-altering decisions every day, often without any human oversight, and almost never with any explanation for why those decisions were made. A 2023 study by the MIT Media Lab found that 68% of large U.S. employers use AI tools to screen entry-level job applications, and that these tools systematically downgrade resumes from women, Black, and Latino applicants by an average of 30% compared to equally qualified white male candidates [^1]. A 2022 investigation by the European Digital Rights network found that 72% of EU companies use AI to screen loan applications, with 41% of those systems showing statistically significant bias against low-income and immigrant applicants [^2]. In the U.S., a 2021 report from the Consumer Financial Protection Bureau (CFPB) found that AI-powered mortgage approval systems rejected 40% more Black applicants with credit scores above 700 than white applicants with identical scores, a disparity that cost Black homebuyers an estimated $3.5 billion in lost wealth between 2019 and 2021 [^3].

For decades, technologists and policymakers have framed AI as a neutral tool: a set of 1s and 0s, unburdened by the implicit biases that shape human decision-making. The pitch has been simple: if we can remove human emotion and prejudice from hiring, lending, healthcare, and criminal justice, we will build a fairer, more efficient world. But Maria’s story, and the thousands of stories like it, expose the lie at the heart of that pitch. Code is not neutral. The data that trains AI is not neutral. The people who design AI systems are not neutral. And the systems we build to govern AI are, as of now, woefully inadequate to stop the harm they cause.

This book is about that harm, and about how we can build AI systems that live up to the promise of fairness, privacy, and human dignity that their creators so often claim. It is built around three core pillars that define the most urgent ethical challenges of the AI age, each explored in depth across the chapters that follow: 1. Algorithmic bias: The systematic, often invisible, unfair treatment of marginalized groups by AI systems, from hiring tools that reject qualified women to healthcare algorithms that deny Black patients the care they need. 2. Digital privacy: The widespread, unregulated collection and use of personal data to train AI systems, and the ways that data is used to manipulate, exploit, and harm the people it is taken from. 3. Lethal autonomous weapons systems (LAWS): AI-powered weapons that can select and attack targets without meaningful human control, raising existential questions about accountability, morality, and the future of war.

These are not abstract problems for tech CEOs or government regulators to solve in closed-door meetings. They are problems that impact every person who uses a smartphone, applies for a job, takes out a loan, goes to the doctor, or lives in a world where AI-powered weapons could one day be used to kill without human intervention. Maria’s story is not just a story about hiring discrimination. It is a story about what happens when we hand over life-altering decisions to systems we do not understand, that we do not regulate, and that we refuse to hold accountable when they fail.

To understand why these failures happen, we first have to answer the question that will guide every chapter of this book: *If code is supposed to be neutral, why does it so often produce unfair outcomes?*

How AI “Learns” to Be Biased: A Primer for Beginners

To understand algorithmic bias, you first have to understand how most modern AI systems are built. The vast majority of AI systems used for decision-making today use a technique called *supervised learning*, which is exactly what it sounds like: you “teach” an AI system by showing it thousands or millions of examples of the decision you want it to make, along with the correct answer for each example.

Let’s use a hiring AI as an example. If a company wants to build an AI that screens job applications, they will first gather a dataset of every resume the company has received for the role in the past 10 years, along with the outcome for each resume: was the applicant hired, and if so, did they stay at the company for at least 2 years (the standard metric for “good performance” in most hiring AI systems). They then feed this dataset into the AI, which uses mathematical formulas to find patterns in the data that correlate with “good” outcomes.

The simplest way to think about this is as a weighted scoring formula. For any given resume, the AI will assign a score $S$ calculated as: $$S = \sum_{i=1}^{n} w_i x_i$$

Where $x_i$ is a feature of the resume (e.g., $x_1$ = gender, $x_2$ = years of experience, $x_3$ = university attended, $x_4$ = zip code) and $w_i$ is the weight the AI assigns to that feature based on the patterns it found in the training data. If the training data shows that 90% of past hires attended a private university, the AI will assign a high positive weight to $x_3$, so applicants who attended a private university get a big score boost. If the training data shows that 80% of past hires were men, the AI will assign a negative weight to $x_1$ for women, so female applicants get a score penalty. If the training data shows that applicants from a certain zip code have a 70% turnover rate, the AI will assign a negative weight to that zip code, even if the zip code is majority Black or Latino, and even if the turnover rate is high because the company has a history of discriminating against employees from those neighborhoods.

This is exactly what happened with Amazon’s now-infamous 2018 hiring AI. Amazon built an AI to screen software engineer resumes, using 10 years of historical hiring data. Because the tech industry has historic