Foundations of AI Security Engineering

1.1 A Question the Industry Has Been Slow to Ask

In the autumn of 2024, a financial technology startup in Singapore lost roughly forty-two thousand dollars in seventeen minutes. The attacker did not break into a database, did not phish a single employee, and never touched a server. They typed a sentence into a chat window on the company's public website, and an autonomous agent — a piece of software that had been explicitly authorized to move money on behalf of customers — cheerfully executed the instruction as if it had been a legitimate withdrawal request. The incident closed quietly, an internal line item labeled "API misuse," and was forgotten within a quarter. A year earlier, a regional hospital's symptom-checking assistant told a worried parent that a common antihistamine, when combined with a frequently prescribed antidepressant, would "almost certainly produce no interaction of clinical significance" — a sentence that, when traced back through the model's output, turned out to have been generated by an internal reasoning pattern that had no clinical knowledge at all, only the statistical shape of reassurance. The parent, fortunately, asked a human pharmacist before administering the combination. Two weeks after the hospital incident, a customer service chatbot operated by a midsize law firm was asked, in apparently innocent English, to summarize its retrieval-augmented knowledge base, and obediently returned a list of internal litigation strategies and, with them, unredacted social-security numbers belonging to seven named plaintiffs.

None of these stories made the front page of a national newspaper. None of them involved a breach in the conventional sense. None of them will appear in the canonical incident reports that classical cybersecurity textbooks have spent the last three decades compiling. And yet each of them is an unambiguous failure of security: of confidentiality in the first, of integrity in the second, of availability paired with catastrophic misinformation in the third. They are not aberrations. They are the predictable, almost scheduled output of an engineering discipline that has been built, shipped, and operated without the vocabulary and tools needed to reason about its own failure modes. The purpose of this chapter is to provide that vocabulary and those tools, and to make the case — patiently and concretely — that the engineering of security for intelligent systems is not a sub-branch of conventional cybersecurity but a separate discipline in its own right, with its own objects of protection, its own adversary models, and its own theoretical commitments.

The argument proceeds in four movements. We will first trace the historical evolution that has produced the current landscape, demonstrating that the attack surface of intelligent systems has expanded in lockstep with their capability. We will then introduce the formal vocabulary required to reason about that attack surface — assets, adversaries, threat surfaces, trust boundaries, residual risk — drawing parallels and contrasts with classical information security so that the genuinely new elements become visible. We will then examine three foundational case studies using a uniform analytical template that will recur throughout this volume, giving the reader an early feel for the texture of AI security work. Finally, we will articulate the engineering mindset required of the practitioner: one grounded in probabilistic rather than deterministic reasoning, committed to defense-in-depth as a default posture, and unwilling to accept security as a property of any single component rather than of the system as a whole. A map of the structural progression of the entire book will then frame the journey on which we are about to embark.

1.2 From Adversarial Examples to Autonomous Agents: A Brief History of the Threat Surface

To understand why AI security is a distinct discipline, it helps to understand how rapidly the field has moved. The young engineer reading this chapter is being asked to enter a discipline whose central object of study has been fundamentally rewritten twice in the space of a single decade. Most other branches of engineering — civil, mechanical, electrical — do not experience such discontinuities. A bridge built in 1980 is governed by the same laws of mechanics as a bridge built in 2020. The object of study is stable. The study of AI security has no such stability, and this fact alone sets it apart.

The first era of what is now called adversarial machine learning began quietly in the early 2010s. Researchers including Christian Szegedy, Ian Goodfellow, and their colleagues demonstrated, between roughly 2013 and 2018, that neural networks trained to high accuracy on natural images could be reliably fooled by imperceptibly small pixel perturbations. An image of a panda, with the right kind of mathematical noise overlaid, would be confidently classified as a gibbon by a state-of-the-art image classifier. The existence of these adversarial examples was, at the time, treated as a curious theoretical nuisance rather than a security problem, in part because the attacks seemed to require white-box knowledge of the model's gradients. The threat surface in this period was narrow: the attacker needed to know a model's parameters, the model had to be a classifier on a fixed input domain, and the manipulation had to occur at inference time. The list of practical exploits was short.

What changed between 2018 and 2023 was not the existence of adversarial behavior but the shape of the systems exhibiting it. The transformer architecture, introduced formally in 2017, made it possible to pretrain large general-purpose models on broad text corpora and then to deploy them across a staggering variety of downstream tasks. By 2022, the practical attack surface of these models extended far beyond the static classification of curated images. Large language models could be steered by natural-language prompts. They could be augmented with retrieval pipelines that fetched context from external corpora at inference time. They could be given tool-calling APIs through which they invoked external functions — sending emails, querying databases, transferring money, executing code. They could be assembled into agentic frameworks that pursued multi-step goals over extended horizons, accumulating state, reading their own outputs, and re-planning as they encountered friction. Each of these capabilities arrived roughly one or two years after the previous, and each arrived accompanied by a corresponding class of attacks. Prompt injection, indirect prompt injection through retrieved documents, function-calling abuse, agent goal hijacking, retrieval poisoning, model exfiltration through output channels — these are not new inventions of malevolent actors but the inevitable consequences of capabilities that were never designed with adversarial environments in mind.

Consider the trajectory from capability to threat in concrete terms. The first generation of consumer-facing chatbots, deployed around 2017 and 2018, accepted a short user query and returned a short response. The surface area for an attack was the query string and the response string. An attacker who could craft a malicious input could trigger one of a small set of failure modes — usually a toxic or off-brand output — that the vendor's post-hoc filtering attempted to catch. By 2020, retrieval-augmented generation had become standard: the model no longer answered only from its own parameters but first consulted an external document store, and the attack surface widened to include whatever documents the system could be induced to retrieve. By 2022, function calling had become standard: the model could issue structured requests to external APIs, and the attack surface widened to include anything those APIs could do. By 2024, agentic frameworks had matured: the model could plan, branch, retry, and persist across many tool calls, and th