Foundations of AI Security Engineering

Foundations of AI Security Engineering

The question at the heart of this book demands immediate attention: what makes AI systems fundamentally different from traditional software from a security perspective? The answer determines everything that follows—how we design defenses, assess risks, and respond to incidents. Traditional software security operates on deterministic principles: given the same input, code either functions as intended or contains a defect that manifests predictably. AI systems, particularly those powered by large language models, operate on fundamentally different principles where the same input may produce varied outputs based on probabilistic inference, context state, and emergent model behaviors that were never explicitly programmed.

This distinction is not merely academic. It represents a categorical shift in how we must approach security engineering. When Google deployed its early spam detection systems in the early 2000s, security engineers could trace every classification decision back to specific model weights and feature inputs. When Microsoft launched Tay in 2016, the AI chatbot that was manipulated through Twitter interactions into generating offensive content within hours, the company discovered that emergent behaviors could circumvent every pre-deployment safeguard they had constructed. The probabilistic nature of machine learning creates attack surfaces that do not exist in traditional software—surfaces where an adversary does not need to exploit a code defect but merely needs to navigate the model's learned decision boundaries in unexpected ways.

Between 2022 and 2024, organizations worldwide experienced this gap between theoretical AI security concerns and practical exploitation. The discovery of prompt injection attacks against deployed language models, the emergence of sophisticated jailbreaking techniques, and the weaponization of AI-powered systems for social engineering demonstrated that the security community's existing frameworks were inadequate. This chapter establishes the conceptual foundation necessary to address these challenges, examining the architectural evolution that created modern AI systems, the novel attack surfaces they introduce, the emergent behaviors that complicate traditional threat models, and the frameworks practitioners need to secure AI-enabled applications.

The Architectural Evolution and Its Security Implications

Understanding why AI systems present unique security challenges requires tracing the architectural evolution from rule-based systems to contemporary neural network architectures. Each transition introduced not only capability improvements but also fundamental changes to the attack surface that security practitioners must address.

From Rule-Based Systems to Machine Learning

Early automated systems, including chatbots and decision-support tools, operated through explicit rule engines. A rule-based customer service bot might follow decision trees such as: if the user types "refund," then display the refund policy and ask for order details; if the user types "track order," then query the shipping database. These systems were security-relevant because each rule represented a potential entry point—an attacker who could manipulate input to traverse unexpected paths might access functionality not intended for their role. However, the attack surface remained fundamentally bounded: the system could only perform actions that developers had explicitly programmed.

The transition to machine learning introduced a paradigm shift that expanded the attack surface in ways that took years for the security community to fully appreciate. Consider the evolution from rule-based spam detection to machine learning classifiers. A rule-based spam filter might check whether an email contained specific keywords, whether the sender appeared on a blocklist, or whether the message structure matched known phishing templates. These rules were explicit and auditable—security engineers could review them, test them against known attack patterns, and verify their correctness. A machine learning spam detector, by contrast, learns to distinguish spam from legitimate email by adjusting millions of internal parameters based on training examples. The resulting model does not encode explicit rules but rather learned representations that correlate with spam characteristics.

This shift created several new security considerations. First, the decision boundary between legitimate and malicious inputs became implicit rather than explicit—security engineers could no longer simply review rules to understand what the system would accept or reject. Second, the model's behavior could vary significantly based on inputs that it had not encountered during training, leading to unexpected classifications that might allow malicious content to bypass detection. Third, attackers discovered that they could systematically probe the model to infer its learned decision boundaries, enabling them to craft inputs that appeared legitimate but were optimized to circumvent the classifier.

The security implications became acutely apparent with the development of adversarial examples—inputs carefully crafted to cause misclassification. Research published in 2013 demonstrated that adding imperceptible perturbations to images could cause state-of-the-art classifiers to confidently misidentify objects. Extending this work to text, researchers showed that modifying a small number of words or characters could allow malware to evade machine learning-based detection systems. These attacks were not exploiting bugs in the code but rather probing the learned decision boundaries of models that had never been explicitly programmed to handle such inputs.

The Transformer Revolution and New Security Dimensions

The introduction of transformer architectures in 2017, and their subsequent scaling to billions of parameters in large language models, represents the most significant architectural evolution in AI history—and the most significant expansion of the AI attack surface. Understanding the security implications of transformers requires examining three core components: the attention mechanism, the massive parameter spaces, and the natural language interface.

The attention mechanism allows transformers to process input text by attending to different portions of the context based on learned relevance. This capability enables the model to maintain coherent conversations over thousands of tokens and to generate contextually appropriate responses. However, it also creates a novel attack surface: the attention patterns themselves can be influenced by carefully crafted inputs. Research has demonstrated that adversarial prompts can manipulate which parts of the context the model attends to, effectively hijacking the model's reasoning process.

Consider how this manifests in practice. An attacker deploying a customer service chatbot might craft inputs designed to manipulate the attention patterns such that the model attends more strongly to injected instructions than to legitimate system prompts. If the system prompt defines behavior boundaries (such as refusing to process payment information), and the attacker's input successfully redirects the model's attention to embedded instructions that override these boundaries, the model may execute actions contrary to its intended behavior. This represents a fundamentally new class of attack that has no direct analog in traditional software security.

The massive parameter spaces of large language models—ranging from billions to hundreds of billions of parameters—create additional security complications. Security through obscurity was never a sound principle, but the scale of modern models makes comprehensive analysis practically impossible. A traditional software system might contain thousands or millions of lines of code that security engineers can theoretically review. A large language model contains