Foundations of AI Security Engineering

Foundations of AI Security Engineering

The fundamental security challenge with AI systems is that they do not behave the way we expect software to behave. When a traditional application receives the same input, it produces the same output—every time, reliably, predictably. When a large language model receives the same input, it may produce a different response on each invocation. This seemingly simple difference in determinism transforms the entire landscape of security, creating attack surfaces that have no analog in conventional software engineering and undermining every assumption that decades of security best practices have been built upon. Understanding why this happens, and what it means for defenders, is the essential starting point for anyone serious about securing AI systems.

This chapter establishes the conceptual foundations for the entire discipline of AI security engineering. We begin by examining what makes AI systems fundamentally different from traditional software, exploring how the probabilistic nature of machine learning outputs creates security challenges that cannot be addressed through conventional practices alone. We then reexamine the CIA triad—Confidentiality, Integrity, and Availability—in the AI context, showing how each pillar of information security must be reconceptualized when the system in question can learn, infer, and generate novel content rather than simply storing and retrieving data. Through detailed examination of historical incidents, from the infamous Tay chatbot poisoning to sophisticated prompt injection attacks, we illustrate the real-world consequences of treating AI systems as if they were just another software component. Finally, we introduce a threat modeling framework specifically designed for AI deployments, giving practitioners a structured approach to identifying and mitigating AI-specific risks.

The Architecture of AI Attack Surfaces

From Deterministic Code to Stochastic Systems

Traditional software security operates on a foundational assumption: code is logic, and logic is deterministic. When you send an HTTP request to a web application, the application executes a predetermined sequence of operations. If the input is well-formed and the code is correct, the output is predictable. Security vulnerabilities in this model—buffer overflows, SQL injection, cross-site scripting—exploit the gap between what the software is supposed to do and what an attacker can make it do by manipulating inputs. But the attack surface is bounded: the code path exists, the vulnerability exists in that code path, and either it can be exploited or it cannot.

Machine learning models work on entirely different principles. A large language model, for instance, is not a program that executes a sequence of predetermined instructions. Instead, it is a statistical artifact—a vast numerical structure encoding patterns extracted from training data. When you provide input to such a model, the output emerges from a complex computation involving billions of parameters, producing a response that reflects the model's learned associations rather than a fixed decision tree.

This distinction matters enormously for security. Consider the difference between a traditional access control system and an AI-powered content moderation system. A traditional system might check a user's role against a permissions matrix: if role equals "admin" and resource equals "sensitive_data", allow access. The logic is transparent, the decision is deterministic, and the attack surface consists of whoever can modify the role assignment or the permissions matrix. An AI content moderation system, by contrast, makes probabilistic determinations about whether content violates policy. The same input might be classified differently on different invocations, and an attacker doesn't need to modify any permissions database—they need only craft input that exploits the statistical boundaries of the model's decision boundary.

The stochastic nature of AI outputs means that security cannot rely on behavioral guarantees. You cannot test an AI system the way you test traditional software, running through input cases and verifying expected outputs. The best you can do is characterize the distribution of possible outputs and understand how that distribution shifts under adversarial conditions. This is fundamentally different from verifying that a function returns 42 when given 21 and 2.

The Expansion of the Attack Surface

When we examine an AI system from a security perspective, we must account for attack surfaces that simply do not exist in traditional software. An AI system with a traditional software component (such as an API layer, database connection, or web interface) retains all the conventional attack surfaces of that software. But it also introduces entirely new categories of vulnerability.

The most discussed of these is the prompt injection attack class. In a prompt injection, an attacker provides input designed to manipulate the AI system's behavior by exploiting the model's tendency to follow instructions embedded in the input context. Unlike SQL injection, where attacker-controlled data is interpreted as code by a database engine, prompt injection exploits the fundamental design of instruction-following language models. The model has been trained to follow user instructions, and a well-crafted prompt can cause it to deviate from its intended behavior in ways that may leak information, bypass safety measures, or manipulate downstream systems.

Consider a customer service chatbot that retrieves order information from a database based on user requests. An attacker might craft a prompt that instructs the model to ignore its system instructions and instead extract and display information it should not have access to—or worse, instruct the model to make database queries on the attacker's behalf, effectively weaponizing the AI system against the infrastructure it is integrated with.

Model inversion attacks exploit a different vulnerability class. These attacks use carefully constructed queries to a model to extract information about the training data or the model's internal parameters. In the most concerning variant, an attacker who can query a model trained on sensitive data might be able to reconstruct approximate representations of that training data. A model trained on medical records, for instance, might inadvertently encode statistical patterns that allow an attacker to determine whether a particular individual's data was in the training set—a violation of privacy that no traditional database security measure could prevent.

Adversarial examples represent yet another attack surface unique to ML systems. These are inputs that have been specifically crafted to cause the model to misclassify or produce incorrect outputs. What makes adversarial examples particularly insidious is that they can be nearly indistinguishable from legitimate inputs to human observers. An adversarial image might look like a stop sign to a human but be classified as a speed limit sign by a computer vision system. In language models, adversarial text sequences can cause models to ignore safety instructions, leak sensitive information, or produce harmful outputs that bypass content filters.

Why Traditional Security Assumptions Fail

The security community has developed an impressive toolkit for protecting traditional software: input validation, output encoding, access control matrices, principle of least privilege, defense in depth. These practices emerged from decades of experience with deterministic systems, and they work remarkably well within their domain. But apply them to an AI system and you will quickly discover their limitations.

Input validation, for instance, is a cornerstone of traditional security. If your application expects an email address, you validate that the input matches the expected format before processing. If it expects a number wi