Foundations of AI Security Engineering

Foundations of AI Security Engineering

The security landscape of modern software systems has undergone a fundamental transformation with the widespread integration of large language models (LLMs) into production applications. Where traditional software security focused on protecting deterministic code paths and static data structures, the introduction of AI-powered systems has created an entirely new category of security challenges that require fundamentally different approaches to identification, assessment, and mitigation. Understanding why AI security engineering constitutes a distinct discipline from conventional software security—and comprehending the specific threat categories that define this field—represents the essential foundation for anyone seeking to secure AI systems in production environments.

This chapter establishes that foundation by examining the historical evolution that led to AI-specific security concerns, analyzing the expanded attack surfaces created by LLM integration, reinterpreting the classical security triad of confidentiality, integrity, and availability in AI contexts, and developing a comprehensive taxonomy of AI security threats. By the conclusion of this chapter, readers will possess the conceptual framework necessary to understand subsequent chapters' detailed explorations of specific attack techniques and defensive architectures.

From Software Security to AI-Specific Security Concerns

The discipline of software security has roots extending back to the earliest days of computing, when researchers first recognized that programs could be manipulated through inputs specifically crafted to cause unintended behavior. The Morris Worm of 1988 marked a pivotal moment, demonstrating how software vulnerabilities could propagate automatically across networked systems with devastating consequences. In the decades that followed, the security community developed increasingly sophisticated understanding of software vulnerabilities—buffer overflows, SQL injection, cross-site scripting, and countless other vulnerability classes—along with corresponding defensive techniques to prevent exploitation.

This historical evolution produced a security paradigm built on several core assumptions that held reasonably well for traditional software systems. First, the assumption of determinism: given the same inputs and the same code, a traditional software system would produce the same outputs, making behavior predictable and testable. Second, the assumption of explicit control: developers retained complete specification over what the system should and should not do, with any deviation from specified behavior representing a bug or vulnerability. Third, the assumption of bounded attack surface: traditional software vulnerabilities existed at identifiable interfaces—network ports, user input fields, file parsers—where attackers could inject malicious data.

The emergence of machine learning as a dominant software paradigm challenged each of these assumptions in fundamental ways. Neural networks, particularly the large language models that now power countless applications, introduce inherent non-determinism through their probabilistic nature. An LLM responding to the same prompt may produce different outputs across invocations due to temperature settings, sampling procedures, or internal state variations. More significantly, the behavior of neural networks emerges from learned parameters rather than explicit developer specification. When a developer writes traditional code, they directly specify the logic that transforms inputs to outputs. When training a neural network, they specify an objective function and provide training data, but the resulting model behavior emerges through optimization processes that humans cannot directly inspect or predict with certainty. This "black box" characteristic means that even the developers who created an AI system may not fully understand why it produces particular outputs.

The attack surface expanded correspondingly. Beyond traditional injection points, attackers gained the ability to manipulate AI systems through specially crafted inputs designed to trigger unexpected behaviors encoded in model parameters. The model itself became an attack surface, with techniques emerging to extract training data, reverse-engineer model architectures, or influence model behavior through adversarial inputs. As organizations began integrating LLMs into applications that handle sensitive data, make consequential decisions, or interact with external systems, the potential impact of security failures grew substantially.

Historical incidents have demonstrated the real-world consequences of these expanded risks. In 2023, security researchers demonstrated that ChatGPT could be manipulated through carefully crafted prompts to ignore its safety guidelines—a vulnerability class now known as prompt injection. Similar research revealed that AI code generation tools could be induced to produce vulnerable code or leak sensitive information from training contexts. These demonstrations, while often conducted by security researchers rather than malicious actors, illustrated that AI systems introduce qualitatively different security challenges requiring dedicated attention.

The recognition of AI security as a distinct field crystallized as practitioners realized that existing security frameworks, while valuable, proved insufficient for addressing AI-specific risks. Traditional security assessments might verify that a web application properly sanitizes database inputs, but they offered limited guidance for evaluating whether an LLM might inadvertently disclose sensitive information from its training data or be manipulated into producing harmful outputs. This recognition motivated the development of AI-specific security disciplines encompassing threat modeling adapted for AI systems, evaluation methodologies designed for ML models, and defensive techniques addressing AI-specific vulnerability classes.

The Expanded Attack Surface in LLM-Integrated Applications

Understanding why AI systems present expanded attack surfaces requires examining the specific ways that LLM integration introduces new vulnerability vectors beyond those present in traditional software. When an organization deploys an LLM-integrated application, multiple components interact in complex ways, each representing potential attack surface that may not have existed in the application's non-AI predecessor.

The most immediately apparent expansion involves user input handling. Traditional applications process user input through well-defined parsing and validation logic. If a web application accepts a username, it might validate that the input contains only allowed characters, falls within length limits, and does not contain SQL injection payloads. These validation rules can be specified precisely and tested exhaustively. When the same application integrates an LLM to process user messages, it must now handle arbitrary natural language input that may contain instructions intended to manipulate the model's behavior rather than merely provide data.

Consider a customer service application that integrates an LLM to generate responses. The LLM processes user messages and generates text-based responses that are delivered back to users. From a security perspective, this integration creates several new attack surfaces. The user-to-model input channel becomes an attack vector where users—or attackers impersonating users—may attempt to inject instructions that manipulate the model into revealing information it should not disclose, generating content that violates policy, or taking actions beyond its intended scope. This channel might have existed previously as user input to a traditional application, but the attack payload now targets an AI model rather than a parser with deterministic behavior.

The model-to-user output channel represents another attack surface. Unlike trad