Introduction to AI Security Engineering

Introduction to AI Security Engineering

The question at the heart of this book is deceptively simple: *How do we secure systems that are inherently unpredictable?* For decades, software security operated on a foundational assumption—that software behaves deterministically. A given input, under the same conditions, produces the same output. This predictability made security engineering tractable: you could enumerate inputs, test edge cases, and verify that your defenses held against known attack patterns. When traditional software failed, it typically failed in ways that were, at least in principle, observable and diagnosable. A buffer overflow corrupts memory in specific ways; an SQL injection manipulates queries according to known grammar rules; a cross-site scripting attack exploits well-defined browser parsing behaviors.

Artificial intelligence systems break this paradigm in fundamental ways. Modern AI applications—particularly those built on large language models—operate probabilistically. They generate outputs based on learned statistical relationships across vast parameter spaces, not by executing predetermined logic paths. The same question asked twice may yield meaningfully different answers. More critically, the boundary between "input" and "code" has blurred in ways that create entirely new vulnerability classes with no direct analogue in traditional software security.

This shift demands a new discipline: AI Security Engineering. This field encompasses the theory, methodology, and practical techniques for identifying, assessing, and mitigating security risks in AI systems. It extends traditional security thinking while acknowledging that AI systems introduce qualitatively different threat models that require novel defensive approaches.

This chapter establishes the foundational understanding necessary for everything that follows. We begin by examining why the probabilistic nature of AI fundamentally changes the security landscape. We then explore the expanded attack surface that emerges when AI systems interact with users, tools, and external data sources. Next, we apply the familiar CIA triad—Confidentiality, Integrity, and Availability—to AI contexts, revealing both alignments with and departures from traditional security thinking. Through examination of historical incidents, we trace how the discipline of AI security has evolved in response to real-world failures. Finally, we map out the structure of this book and explain how each subsequent chapter builds upon these foundations to develop a comprehensive security mindset for AI practitioners.

---

The Paradigm Shift: From Deterministic to Probabilistic Computing

Understanding Deterministic Security Assumptions

Traditional software security operates on assumptions that have proven remarkably durable since the early days of computing. When you authenticate a user, you execute a specific comparison operation: does the submitted password hash match the stored hash? The logic is unambiguous. When you validate input, you check whether characters conform to expected patterns using regular expressions or parsing logic with discrete branches. When you authorize access, you evaluate boolean conditions: does this user have the required role? Does the resource belong to this tenant? The system transitions between well-defined states, and every operation has a specification against which correctness can be verified.

This determinism enables security analysis in ways that feel natural to experienced practitioners. Attackers probe for buffer overflows by sending inputs of specific lengths; defenders respond by enforcing bounds checking. Attackers manipulate SQL queries through character injection; defenders respond with parameterized queries. The pattern-remediation cycle works because vulnerabilities follow comprehensible rules. Buffer overflows exist because C permits writing past array boundaries. SQL injection exists because SQL concatenation blends code and data. Each vulnerability has a root cause that can be understood, fixed, and prevented through principled coding practices.

Penetration testing and security auditing derive their power from this determinism. When a security researcher discovers that sending a specific sequence of bytes causes a server to crash, that finding is reproducible and actionable. The development team can reproduce the crash, identify the vulnerable code path, and implement a targeted fix. Security becomes a tractable engineering problem because failure modes are bounded and observable.

The Probabilistic Nature of AI Models

Modern AI systems, particularly deep learning models and large language models, operate on fundamentally different principles. These systems learn statistical representations from training data, encoding patterns as numerical parameters in neural network weights. When presented with an input, the model doesn't "look up" an answer—it computes a response by propagating activations through billions of parameters, transforming the input through numerous non-linear transformations until producing an output distribution.

Consider the difference concretely. A traditional spam filter might work by checking whether an email contains known spam keywords, whether the sender's IP address appears on reputation lists, and whether the email structure matches known phishing patterns. The decision logic is explicit and auditable. An AI-based spam filter, by contrast, learns to recognize spam through exposure to millions of examples. It develops internal representations of what "spam" looks like in a high-dimensional feature space, where each email is represented as a point and spam is distinguished from legitimate mail by learned decision boundaries. The filter doesn't have an explicit list of spam keywords—it has a continuous mapping from input space to output space that emerged from optimization on training data.

This probabilistic foundation introduces several security-relevant properties that have no direct analogue in traditional software:

Gradient-based sensitivity: Neural networks are continuous functions approximated by discrete computation. The same semantic input phrased slightly differently—using synonyms, reordering clauses, or adding seemingly irrelevant context—can shift the model's output in ways that are difficult to predict without actually running the model. This sensitivity creates attack surface: adversaries can craft inputs that exploit this continuity to steer model behavior.

Emergent capabilities and failure modes: Large language models exhibit capabilities that weren't explicitly programmed but emerged from scale—reasoning, translation, code generation. They also exhibit failure modes that emerge unpredictably: the tendency to "hallucinate" plausible but incorrect information, susceptibility to prompt injection, and the potential to produce harmful outputs when prompted in specific ways. These emergent properties make comprehensive testing infeasible; you cannot enumerate all possible inputs to verify correct behavior.

Opacity of decision processes: While explainability research has made progress, neural networks remain fundamentally opaque. When a traditional system makes a security-relevant decision—blocking a transaction, flagging content, or granting access—the decision can be traced through explicit logic. When a neural network makes a similar decision, the reasoning is distributed across billions of parameters in ways that resist human interpretation. This opacity complicates both security auditing and accountability.

Stochastic sampling introduces unpredictability: Most language models generate outputs through sampling from a probability distribution over tokens. The same input processed twice may yield different outputs due to random sampling decisions. While sampling temperature and other parameters can constrain this variability, they cannot eliminate it. Security decisions