Foundations of AI Safety Engineering

The Probabilistic Trust Problem

Securing a system whose core component is a probabilistic natural-language model demands a discipline that traditional information security does not yet possess. Classical application security rests on deterministic abstractions: inputs are parsed against grammars, code paths are exercised by tests, and a successful attack requires a flaw that can be located in a finite, inspectable artifact. A large language model collapses several of these abstractions simultaneously. The same artifact — a sequence of tokens — serves as the user interface, the instruction channel, the application logic, and the trust boundary. The model itself is a statistical function approximator whose outputs are sampled from a distribution rather than computed from a specification. The system, viewed end to end, behaves less like an API and more like a runtime that interprets an untrusted programming language in which the source code is indistinguishable from prose. The central question of this book is therefore narrow and consequential: given such a system, what does it mean, as an engineering discipline, to "secure" it, and how does that discipline relate to — and differ from — classical information security and from the emerging science of AI alignment?

AI Safety Engineering: A Working Definition

We define *AI safety engineering* as the systematic identification, modeling, mitigation, and verification of risks that arise when generative models participate in production systems as decision-makers, reasoners, or tool invokers. The definition has four deliberate components. *Identification* is the discovery of plausible failure modes, drawing on taxonomies such as the OWASP LLM Top Ten, MITRE ATLAS, and the NIST AI Risk Management Framework. *Modeling* is the construction of artifacts — typically threat models, abuse cases, and trust diagrams — that make those failure modes falsifiable and reviewable. *Mitigation* is the design of controls whose effectiveness can be argued from the model. *Verification* is the empirical and formal demonstration that the controls hold against relevant adversaries.

The discipline is engineering rather than science because it is constrained by the need to ship systems that operate on a fixed schedule under bounded cost. It is safety engineering rather than security engineering alone because the harms it must prevent include not only intentional adversarial behavior but also unintentional harm arising from model brittleness, distributional drift, and the structural mismatch between training objectives and deployment objectives. It is generative-system-specific rather than generally AI-focused because the failure modes introduced by autoregressive token sampling, instruction following, in-context retrieval, and tool use have no clean analogue in classical machine learning pipelines.

A working definition must also specify what is *not* in scope. AI safety engineering does not attempt to define the values the model should express — that question belongs to alignment research and AI ethics. It does not attempt to certify that a model is "aligned" in any philosophical sense; it accepts whatever model the organization has chosen to deploy and treats the model's behavior as a partially controllable black box. It does not, in this book, address long-tail catastrophic risks such as deceptive power-seeking behavior, except insofar as such risks have analogues in production systems (for example, agentic systems that pursue mis-specified objectives through tool abuse). The discipline is pragmatic: it operates where probability distributions meet production code.

Disciplinary Boundaries: How AI Safety Engineering Differs

The field sits at the intersection of three established disciplines and inherits obligations from each while differing in emphasis from each. Understanding the boundaries is necessary because borrowed vocabulary can mislead.

*Application security* provides the foundational methodology: threat modeling, trust boundaries, defense in depth, the STRIDE taxonomy, and the principle of least privilege. AI safety engineering inherits all of these. It differs from classical application security in that the trust boundary it must defend is not a network perimeter or a process boundary but a token stream. An attacker does not need to exploit a buffer overflow; they need only produce a string that, when concatenated with a system prompt and forwarded to a model, elicits a behavior the system was designed to prevent. The discipline therefore extends classical threat modeling with new attacker primitives: prompt injection, indirect prompt injection via retrieved documents, embedding inversion, jailbreaking, and tool-mediated exfiltration.

*AI alignment research* provides the scientific background: the study of how training objectives, fine-tuning procedures, and constitutional methods shape model behavior. AI safety engineering inherits from alignment a sober view of what models can and cannot be made to do reliably. It differs from alignment research in its temporal horizon and its unit of accountability. Alignment research tolerates open-ended investigation with no delivery deadline; safety engineering must produce a system whose residual risk is acceptable to a release manager on a specific date. Alignment research may conclude that a behavior is "difficult to elicit"; safety engineering must either deploy a control that prevents elicitation or accept the residual risk and document it. The two disciplines are complementary: alignment findings feed into safety controls, and safety incidents feed back into alignment research priorities.

*AI ethics* provides the normative scaffolding: principles such as fairness, transparency, and accountability. AI safety engineering inherits the obligation to consider harms beyond confidentiality and integrity. It differs from AI ethics in that it expresses those principles as operational requirements rather than as policy statements. "The system should not discriminate" is an ethical requirement; "the system must reject loan-application explanations whose reasoning trace contains a protected attribute as a deciding factor" is a safety requirement. The translation from the former to the latter is the central operational task.

The relationship is summarized in the table below.

| Discipline | Primary Unit of Work | Time Horizon | Failure Mode of Concern | |------------|---------------------|--------------|-------------------------| | Application security | Code and infrastructure | Release cycle | Unauthorized access, code execution | | AI alignment | Model weights, training | Research years | Spec gaming, deception, value drift | | AI ethics | Principles, policies | Policy cycle | Discrimination, opacity, exclusion | | AI safety engineering | System with embedded model | Release cycle | Probabilistic failure of model-mediated trust |

The discipline is distinct from each of its neighbors and yet depends on all three. Practitioners who treat AI safety engineering as a subfield of application security will underweight behavioral risks. Practitioners who treat it as a subfield of alignment research will miss the engineering pragmatics. Practitioners who treat it as applied AI ethics will produce documents rather than systems.

The Untrusted Code Interpreter Thesis

The first foundational argument of this book is that a language model embedded in a production system is best understood as an *untrusted code interpreter* rather than as a deterministic API. The argument has three components: structural, behavioral, and operational.

*Structurally*, a language model accepts an input — typically a concatenation of system instructions, retrieved context, tool outputs, and user text — and produces an output that may include free-form text, structured calls to external functions, or both. The set of programs the model can be induced to execute is not enumerated i