Why AI Safety Matters (Even If You Are Not a Coder)

Why AI Safety Matters (Even If You Are Not a Coder)

A few months ago, a customer opened a chat window on a popular airline's website to ask about a refund on a bereavement fare. Within ninety seconds, an "AI assistant" had helpfully suggested that the customer email their full name, address, date of birth, and the sixteen digits on the front of their credit card to a Gmail address that the assistant invented on the spot. The address didn't belong to anyone at the airline. It didn't really belong to anyone. The AI had hallucinated it from whole cloth, the way a sleep-deprived intern might invent a phone number rather than admit they didn't know the real one. The customer, trusting the chatbot because the chatbot was the airline's official interface, sent the information. We can only guess what happened next. The airline learned about the problem from a security researcher who reported it publicly, not from its own monitoring.

That story is not unique. It is not even rare. It is the kind of thing that happens every week, somewhere in the world, to someone who just wanted to do a simple task and trusted a computer to help them. And it is one of the reasons I wrote this book.

You are probably not a machine learning researcher. You may not be able to write code at all. You might be a project manager, a teacher, a small business owner, a parent, a lawyer, a doctor, a journalist, a student, or simply someone who uses the internet and wonders whether all this AI stuff is going to be okay. This book is for you. AI safety is not a topic reserved for people with PhDs or people who work in Silicon Valley research labs. It is a topic that affects every single person who uses a computer, a phone, or a service that quietly runs an AI in the background — which, by 2025, is almost everyone on Earth who has ever touched a screen.

This opening chapter answers one simple question: why should you, an ordinary human, care about AI safety? We are going to look at three real-world failures that made the news (or should have), I'm going to explain in plain language what large language models actually do and why they are fundamentally different from every other piece of software you've ever used, and I'm going to introduce the three categories of failure that this entire book is built around. By the end of the chapter, you will be able to explain to a friend over coffee what AI safety actually means, why it is different from "cybersecurity" in ways that matter, and which three flavors of AI mishap we are going to spend the rest of the book learning to defend against.

No prior knowledge required. Bring your common sense. Leave the jargon at the door.

Real Failures, Real People

Let me start with stories, because stories are how humans learn anything that actually sticks. Three short ones, drawn from public reports, lawsuits, and the kind of post-mortem blog posts that security researchers publish when they have stopped being angry and started being helpful. I have simplified them a little — names of companies removed where appropriate, dates blurred, some technical detail shaved off — but the shape of each failure is exactly as it happened. If anything, I have made them sound slightly less weird than the originals.

Story one: the chatbot that promised refunds it could not deliver.

In 2023, Air Canada's customer service chatbot was asked by a passenger how to claim bereavement-related refunds. The chatbot, in a moment of confident invention, told the customer that they could book a regular ticket and then request a refund after the fact through a special link in the airline's policy page. The customer did exactly that, was denied the refund, and asked Air Canada to honor what its own assistant had said. Air Canada argued in court that the chatbot was a "separate legal entity" and therefore not responsible for its promises. A Canadian tribunal disagreed, ordered the airline to pay up, and the airline's reputation took a hit that no algorithm can fix. The lesson here is not that Air Canada has bad AI. The lesson is that any system that talks to your customers is, for legal and moral purposes, *speaking for you*, even when its words were generated one token at a time by a probability model that did not understand what it was saying. The chatbot did not "intend" to deceive. It simply produced the most statistically plausible continuation of the conversation, and that continuation happened to be wrong. The customer had no way to tell the difference.

Story two: the medical assistant that invented a drug dose.

In early 2024, a researcher asked a popular medical AI assistant a question about a real drug interaction. The assistant replied, with calm authority, that a specific over-the-counter pain reliever could be safely combined with a specific blood thinner at a dose that would, in reality, put an average adult at significant risk of internal bleeding. The researcher noticed the error. A patient reading the same response might not have. There was no warning label, no "I am not a doctor," no little flag in the corner. Just confident, fluent, life-threatening nonsense, formatted in tidy paragraphs as if it had been lifted from a medical textbook. The system had been trained on a corpus of medical text that contained the names of both drugs. It had not been trained to *understand* the dose-response relationship between them. It just knew that the words "safe to combine" sometimes appeared near those drug names in similar sentences. When in doubt, it filled in the gap with the most statistically plausible continuation. The result read like an answer from a kind, knowledgeable doctor. It was, in fact, the output of a very expensive pattern matcher pretending to be a doctor.

Story three: the coding assistant that wiped a production database.

This one came from a developer who posted about it on a developer forum (and later, more cautiously, on a podcast). They were using an AI coding agent — one of those tools that can edit files, run shell commands, and generally act like a junior programmer with extremely fast typing and no sense of consequence. They asked the agent to clean up some unused files on a server. The agent, with the cheerful obedience of an intern who wants to be helpful, interpreted "clean up" rather broadly. It ran a command intended to delete files but, due to a subtle misreading of the directory structure, executed the deletion on the *production* database directory. The cleanup worked. So did the deletion. The database was unrecoverable from backups because, ironically, the agent had also helped "optimize" the backup scripts the week before. The company was offline for two days. The developer was unemployed for considerably longer. The total cost, including lost revenue, incident response, and customer compensation, was reportedly in the seven figures. The cost was not paid by the AI, which does not have a bank account. The cost was paid by the humans who had given the AI more access than they should have.

Three stories. Three different industries — travel, healthcare, software. Three entirely different flavors of failure. And yet, when you look closely, you can see the same underlying issue: AI systems that *sound* like they understand, but are actually doing something closer to very confident guesswork. The chatbot did not know it was lying. The medical assistant did not know it was dangerous. The coding agent did not know it was destructive. None of them would have raised its virtual hand and said "wait, I don't actually know this, please ask a human." That gap — between what the model sounds like it knows and what it actually knows — is the gap we are going to spend this book closing.

Let me pause here to say something important. This is not an anti-AI book. AI is, on the whole, a remarkable technology. It helps people write, code, learn, design, and create in ways that would have seemed like science fiction a decade ago. Many of the