The Decision: When Machines Judge

The rejection letter arrived on a Tuesday, in a plain white envelope that looked no different from any other piece of mail. Sarah Chen had been expecting an answer about her loan application for weeks, checking the mailbox each evening with a mixture of hope and dread that had become as familiar to her as the morning drive to work. She stood on her front porch in the October chill, the envelope still sealed, trying to remember the last time she had felt this anxious about anything. The mortgage on the small craftsman house she had lived in for fifteen years was coming up for renewal, and the bank had told her they needed to review her application through their new system before they could proceed.

Inside, she set her bag down by the door and walked to the kitchen table, the one where she had helped her daughter Maya with homework for years, where she graded papers on Sunday mornings with coffee growing cold beside her. She opened the envelope with careful fingers, as if the words inside might change depending on how roughly she treated the paper. The letter was brief and impersonal, the kind of document that could have been generated for anyone, because in a sense, it was. It informed her that her application had been reviewed and that, based on the information provided, the bank was unable to approve her request at this time. There was no explanation of what information had been reviewed, no details about which factors had weighed against her, no avenue for appeal beyond a generic customer service number that she knew from experience would be staffed by representatives who had no power to change a decision made by the system upstairs.

Sarah read the letter three times, looking for something she might have missed, some reason she could understand and therefore address. She was fifty-three years old. She had taught seventh-grade science at Westbrook Middle School for twenty-one years. She had paid her mortgage on time every single month since she bought the house in 2009, weathering two salary freezes, a global pandemic, and the gradual erosion of her savings as medical bills accumulated for her mother's care. Her debt-to-income ratio, as far as she understood such things, had always been reasonable. She had credit scores that she had checked herself, scores that seemed respectable if not exceptional. What could possibly have gone wrong?

She called the customer service number and waited for forty-seven minutes. When a representative finally answered, a young man with a slight static in his headset who sounded like he was calling from a basement somewhere, he confirmed that her application had been processed through their automated system but could not tell her which specific factors had led to the denial. When she asked to speak to someone who could explain or review the decision, she was told that the system was designed to be comprehensive and consistent, and that human review of automated decisions was generally reserved for cases involving fraud or special circumstances. He apologized for the inconvenience and read her a script about alternative financial products she might want to consider.

Sarah hung up the phone and sat at her kitchen table for a long time, watching the light change outside her window as afternoon softened into evening. She had always believed, in some vague and unexamined way, that decisions about money and credit were made by people—people who would look at her history, her circumstances, her story, and make a judgment based on whether she seemed trustworthy and reliable. That belief had survived decades of paperwork and bureaucracy, survived the indignities of credit checks and background verifications, because at the end of every process there had always been a person, or at least the possibility of a person, who could listen and understand and make an exception if the situation warranted it. Now that belief felt suddenly fragile, like a shelf that had held her weight for years and was only now revealing the slight wobble that had always been there.

This is a story about Sarah Chen, and about millions of people like her whose lives are increasingly touched by decisions made by systems they cannot see, cannot understand, and cannot question. It is about the quiet revolution that has transformed the way decisions get made in banking and hiring, in healthcare and criminal justice, in the thousand small moments where an algorithm decides what you get and what you don't, who matters and who doesn't. And it is about the question that Sarah would spend the coming months trying to answer, a question that has no easy resolution but that demands our collective attention: when machines judge us, what happens to our humanity?

---

The Invisible Adjudicator

To understand what happened to Sarah, we need to understand something about the systems that now govern so many aspects of our lives. These systems go by many names—machine learning algorithms, artificial intelligence, predictive models—but at their core, they share a common purpose: to make decisions faster, cheaper, and more consistently than human beings ever could. In the banking industry alone, algorithmic decision-making now determines who gets loans and at what interest rates, who gets approved for credit cards and credit limits, who gets flagged for fraud and who gets locked out of their own accounts. A 2020 survey by the National Consumer Law Center found that 73 percent of large financial institutions used some form of automated decision-making in their lending processes, a number that had grown from 48 percent just five years earlier. By 2024, that figure had climbed above 85 percent, and the systems themselves had grown far more sophisticated.

The basic idea behind most of these systems can be illustrated with an analogy that Sarah herself might recognize, because she had used it countless times in her own classroom. Imagine a child learning to distinguish between cats and dogs. You don't sit the child down and explain the biological classification of felids versus canids, the structural differences in skull shape and dental arrangement. Instead, you show the child many examples of cats and many examples of dogs, and eventually the child develops an intuition about what makes something a cat rather than a dog. The child learns to notice certain features—the shape of the ears, the behavior of the animal, the texture of the fur—and to weigh those features against each other in ways that are difficult to articulate but that become remarkably accurate with enough practice.

Machine learning works on the same principle. A system designed to predict credit risk doesn't receive a list of rules like "reject anyone with a debt-to-income ratio above X." Instead, it is given millions of examples of past lending decisions, including information about the applicants and information about whether those applicants eventually defaulted on their loans or paid them back faithfully. The system then looks for patterns in these examples, patterns that might be invisible to human observers but that the system learns to recognize and weight in its own opaque way. When a new application comes in, the system compares it to the patterns it has learned and produces a prediction: this applicant looks more like the people who defaulted, or more like the people who paid back their loans.

This process of learning from examples is extraordinarily powerful. It allows systems to find subtle relationships in data that no human analyst would have the time or cognitive capacity to discover. It allows decisions to be made at a scale and speed that would be impossible for any team of human reviewers. And in many cases, it produces outcomes that are genuinely better than human judgment—more accurate, more consistent, less subject to the biases and blind spots that infect human decision-making.

But there is a cost to this power, and it is a cost that Sarah discovered when she opened that envelope