Foundations of AI Security Engineering

Foundations of AI Security Engineering

The question that animates this entire book is deceptively simple: why do the security principles that have protected software systems for decades suddenly seem inadequate when applied to AI systems? The answer lies in a fundamental reorientation of what "computation" means. Traditional software executes precise instructions deterministically—given the same input, it produces the same output, every time. An AI system, particularly one built on large language models, operates probabilistically. It generates responses based on learned patterns, adapting its behavior based on context in ways that are powerful but also unpredictable in ways that have no precedent in classical computing. This probabilistic nature, combined with the flexibility of natural language interfaces and the complexity of integrating AI components with traditional software systems, creates a threat landscape that requires new conceptual frameworks and new defensive strategies.

Understanding AI security requires abandoning the comfortable assumption that inputs map cleanly to outputs through a well-understood transformation. In traditional software, a buffer overflow is a buffer overflow—well-defined, well-studied, and addressed through well-established mitigations. In an AI system, the equivalent "overflow" might manifest as a seemingly innocent phrase that causes the model to reveal sensitive training data, or a carefully crafted prompt that bypasses safety guardrails while appearing completely benign to human reviewers. The attack surface is natural language itself, and natural language is infinitely expressive in ways that structured data inputs are not. This chapter establishes the foundational concepts you need to reason about these novel threats and introduces a threat modeling framework specifically designed for AI systems.

The Convergence of Two Disciplines

AI security is not cybersecurity applied to AI systems—it is something new that emerges from the intersection of two complex disciplines, each with its own vocabulary, assumptions, and defensive traditions. To understand AI security, you must first understand what each parent discipline brings to the intersection and where their assumptions conflict.

Cybersecurity operates on the principle of controlled access. Traditional security assumes that if you can control who accesses what, when, and how, you can build defensible systems. The classic security triad—confidentiality, integrity, and availability—assumes that data and systems have stable, identifiable states. A file is either encrypted or not; a system is either authorized or not; a transaction either completes successfully or fails. Attackers exploit violations of these clear boundaries, and defenders build walls, checkpoints, and monitoring systems to maintain them.

Machine learning operates on the principle of pattern recognition. ML systems learn statistical relationships from data, and their "behavior" emerges from these learned patterns rather than from explicit programming. This creates a fundamentally different kind of system—one whose outputs are determined by both its inputs and its training history, often in ways that are opaque even to their creators. The ML community has long grappled with concepts like generalization, overfitting, and adversarial fragility, but these have traditionally been framed as performance or reliability issues rather than security issues.

When these disciplines converge, their assumptions collide. Consider the problem of input validation. In traditional software, input validation means checking that data conforms to expected formats—strings are within length limits, numbers are in valid ranges, and structured data matches schemas. The validation is deterministic: either the input passes or it fails. In an AI system, input validation becomes radically different. A prompt injection attack presents inputs that are syntactically valid, semantically reasonable, and completely harmless in isolation—but that cause the system to behave in unintended ways when processed in context. How do you validate that a user's question about recipe modifications won't cause your customer service chatbot to reveal private account information? The input isn't "invalid"—it's valid in all the ways traditional validation checks, yet dangerous in ways that traditional validation cannot detect.

This convergence creates what we might call the probabilistic attack surface. Every AI system has a set of intended behaviors—the responses it should produce, the actions it should take, the boundaries it should maintain. But because AI systems operate probabilistically, there is always a non-zero probability that they will produce something outside this intended range. Skilled attackers learn to shift these probabilities, making unintended behaviors more likely through careful input manipulation. This is fundamentally different from traditional software attacks, which typically exploit deterministic bugs. You cannot "patch" a probabilistic system the way you patch a buffer overflow—you can only shift the probability distributions that govern its behavior.

Another critical difference lies in the concept of trust. Traditional systems maintain clear trust boundaries: the inside is trusted, the outside is untrusted, and interactions across the boundary are carefully mediated. Network firewalls create perimeter trust; authentication systems establish user trust; encryption protects data in untrusted environments. These boundaries are discrete and enforceable through technical controls.

Trust in AI systems is far more fluid. An AI model learns from data, and that data may come from users, third-party services, or the internet—all untrusted sources. The model internalizes patterns from this data and applies them in ways that may not be easily separable from its core functionality. When a model trained on internet text encounters a prompt that resembles content it has seen during training, it may reproduce that content in ways that violate the intended trust relationship. More fundamentally, the model itself becomes a component whose trust properties are difficult to specify and verify. Traditional software is trusted to do exactly what its code specifies; an AI model is trusted to behave appropriately across a vast space of possible inputs—inputs that cannot be enumerated in advance because they include natural language expressions that are fundamentally open-ended.

The practical implication of this convergence is that AI security practitioners must be fluent in both domains. You need to understand traditional security controls—authentication, authorization, input validation, output sanitization, logging, monitoring—because AI systems are deployed within software architectures where these controls remain relevant. But you also need to understand the ML-specific attack surface: training data poisoning, model extraction, adversarial examples, prompt injection, and the emergent behaviors that arise from complex AI system architectures. The defender who knows only cybersecurity will miss the AI-specific attack vectors; the defender who knows only machine learning will not have the conceptual tools to integrate security into system design.

Trust Boundaries in AI Architecture

To reason effectively about AI security, you must first understand where trust boundaries exist in AI systems and how they differ from the boundaries in traditional software. A trust boundary is a line across which you assume that one side is more trustworthy than the other, requiring verification before accepting data or commands from the less-trusted side. In traditional web applications, trust boundaries typically exist at the network perimeter, at authentication checkpoints, at database queries, and at integration points with external services. These boundaries are relatively stable and well-defined.

AI systems introduce