Foundations of AI Safety Engineering

Chapter 1 — Foundations of AI Safety Engineering

1.1 Why "Just Align the Model" Is Not an Engineering Answer

Three months after launching a public-facing RAG-based assistant, a mid-sized e-commerce company discovered that a carefully crafted customer email could trick the system into disclosing another customer's order history, including email addresses and partial credit card digits. Postmortem analysis revealed that the underlying foundation model was not "broken" in any meaningful sense — RLHF training had reduced harmful outputs dramatically, refusal rates on direct adversarial prompts were over 99 percent, and the model passed every standard red-team benchmark the vendor had published. The vulnerability lived at the seams: a retrieval pipeline that passed untrusted text directly into the prompt, a memory layer that stored prior user turns without provenance tags, and an output formatter that trusted the model's prose in ways the model had never been trained to handle.

This incident is representative. Across surveys of production LLM deployments in 2024 and 2025 — including the OWASP incident reports, the GenAI Red Team network's quarterly briefs, and the case collections from the Cloud Security Alliance — the dominant pattern is not model malalignment in the alignment-research sense, but brittle composition. The model's behavior, considered in isolation, is usually acceptable. The damage emerges from how the model is *wired into* the rest of the system: what it is shown, what it is asked, what it is allowed to do, and what is done with its output. Alignment research asks how to make the model itself behave well across the full distribution of potential inputs. Defensive AI Engineering asks how to make a *system that contains* the model behave well, in a specific deployment context, against an adversary who knows the model is there and is actively probing for seams.

This distinction is not pedantic. It dictates who is responsible for safety, what methods are appropriate, and what counts as a complete answer. If safety is an alignment problem, then the only mitigation is "better training," and the only responsible actor is the model developer. If safety is an engineering problem, then mitigation is a set of composable controls applied at multiple architectural layers, and the responsible actor is the team that built and operates the system — which is to say, almost everyone reading this book. The chapters that follow treat AI safety as an engineering discipline. This chapter lays the conceptual foundation: a vocabulary, a taxonomy, a threat model, a community baseline, a methodology, and a case study that we will return to throughout the book.

1.2 Historical Threads: From Autonomy Safety to LLM Operations

The engineering discipline that this book belongs to did not begin with language models. Its intellectual lineage runs through autonomous-vehicle safety, reinforcement-learning safety, classical cybersecurity, and safety-critical systems engineering — each of which contributes tools, failure taxonomies, and design patterns that survive the transition into the LLM era. Understanding the lineage matters because it explains why practitioners from each of these fields recognize some of the patterns in AI safety incidents immediately, while being confused by others.

Autonomous-vehicle safety contributed the concept of an *Operational Design Domain* (ODD): the formal specification of conditions under which a system is expected to operate safely, and the corresponding acknowledgment that outside the ODD, behavior is undefined. The ODD concept translates to LLM systems almost verbatim: every production system has an intended use — "answer questions about our product catalog," "summarize legal contracts," "triage IT tickets" — and the safety case must hold inside that envelope. The minute a user asks the customer-support agent "what do you really think of my manager?" the system is operating outside ODD, and an engineering response (deflect, log, refuse) is more reliable than a research-style response (try to train the model to refuse more such queries).

Reinforcement-learning safety contributed a second concept that has aged remarkably well: *untrusted state in the observation channel*. Classic RL agents trained in simulation frequently failed when deployed in the real world because the sensor stream contained adversarial or distributionally shifted inputs. The lesson — that an agent making decisions on the basis of an observation should treat that observation as untrusted even when it comes from a nominally trusted sensor — is exactly the lesson of prompt injection. A user-supplied email that becomes part of a RAG context is, from the model's perspective, an untrusted observation. Treating it as trusted is the same class of error as treating a depth-camera feed as authoritative when an adversary can paste stickers on the camera.

Classical cybersecurity contributed threat-modeling frameworks — STRIDE, PASTA, attack trees — and the trust-boundary abstraction itself. A trust boundary is a line across which data and control cross from a less-trusted zone to a more-trusted zone, and at which a defensive control is required. LLM systems are dense with trust boundaries. The user's prompt crosses into the system; retrieved documents from the open web cross into the context; tool outputs cross back into the model's reasoning; the model's prose crosses out to the user and to downstream automations. Each crossing is a place where defenses must be inserted. The cybersecurity lineage also contributes the vocabularies of confidentiality, integrity, and availability, which we will extend below.

Safety-critical systems engineering — the discipline governing avionics, medical devices, and nuclear control rooms — contributes the most demanding concept: *defense-in-depth*. The argument is that no single control can be assumed to work. The system must be designed so that if control A fails, controls B, C, and D still catch the failure. In an avionics system, this means redundant sensors, redundant flight computers, and a pilot. In an LLM system, it means input filtering plus output filtering plus constrained tool execution plus rate limiting plus human review — not because each is expensive to add, but because any one of them can fail.

Where AI safety historically diverged from these lineages — and the divergence is what this book is partly about correcting — is in its tendency to treat safety as a property of the model alone, optimizable during training rather than engineered around deployment. The reasons for this divergence are real: foundation models are unusually general-purpose, the failure modes are unusually fluent (a bad output looks grammatical), and the safety mechanisms that ship with the model are unusually opaque. But the cost of the divergence is now visible in incident data. The remainder of this chapter introduces the conceptual scaffolding — taxonomy, threat model, methodology — that lets us move safety back into the engineering register.

1.3 Two Layers, Two Languages: Model-Layer Alignment vs System-Layer Safety

A practitioner building an LLM application is responsible for a system, not a model. The system contains the model, but it also contains retrieval pipelines, memory stores, tool runtimes, identity layers, rate limiters, loggers, and downstream automations. Failures of safety can be classified by the layer at which they originate, and the distinction matters because the available mitigations differ at each layer.

Model-layer risks are properties of the model itself. They include: the model can be coaxed into generating harmful content through carefully constructed prompts (jailbreak susceptibility); the model hallucinates facts with high confidence (epistemic-calibration failures); the model exhibits undesirable behavior at the model level even when prompted benignly (training-distribution failure); the model can be