The AI Security Landscape: Threats, Vulnerabilities, and Engineering Imperatives

The AI Security Landscape: Threats, Vulnerabilities, and Engineering Imperatives

The security posture of an organization changes fundamentally when large language models become intermediaries between users and system actions. Traditional software operates through deterministic code paths: input arrives, validation occurs, transformation executes, and output emerges through mechanisms developers can trace, inspect, and verify. The attack surface exists within knowable boundaries—SQL queries can be logged, API calls can be monitored, memory access patterns can be audited. But when an LLM enters the architecture, this predictability dissolves. The model does not execute code in any conventional sense; it generates text based on patterns learned from training data, responding to inputs with outputs that are probabilistically determined and contextually dependent. This transformation raises a question that the security community is only beginning to answer with rigor: how do we secure systems when the core processing element is a learned representation that neither developers nor security tools can fully inspect, predict, or control?

This chapter establishes the foundational context for understanding AI security engineering by examining the structural differences between traditional software and LLM-integrated applications. We explore how the *interpretability gap*—the inherent uncertainty in understanding model behavior—creates attack surfaces that existing frameworks fail to address. Through analysis of real incidents and systematic threat categorization, we develop an adapted security framework suitable for the unique characteristics of AI systems. The engineering imperatives that emerge from this analysis will guide the defensive strategies detailed throughout the remainder of this book.

The Dissolution of Traditional Boundaries

Understanding why AI security requires fundamentally different approaches begins with examining how integration with LLMs transforms the architecture of software systems. In a conventional web application, data flows through well-defined boundaries: the frontend receives user input, the backend validates and sanitizes that input, business logic executes against a database, and responses return through the same validated channels. Each boundary represents a point where security controls can be applied. Input validation can check for type correctness, length limits, and prohibited characters. Authentication layers verify user identity before granting access to protected resources. Authorization checks ensure that even authenticated users can only access resources appropriate to their privileges. These controls work because the system's behavior can be specified completely—what is allowed, what is forbidden, what constitutes valid input.

LLM integration disrupts this model by inserting a component that processes natural language rather than structured data, produces outputs that cannot be predetermined from inputs alone, and operates through learned representations rather than explicit logic. Consider a customer service chatbot that uses an LLM to generate responses. A user types a question; the LLM processes the text, consults its internal weights, and produces a response. The response might include information from the organization's knowledge base, might execute actions through API calls, might access customer data, or might do none of these things in ways that match developer expectations. The traditional security boundary between "input" and "output" becomes permeable when the model serves as an interpretive layer that transforms user intent into system actions.

This dissolution of boundaries manifests in several concrete ways that create novel attack surfaces. First, the model itself becomes part of the trust boundary. In traditional systems, trust is established through authentication tokens, session cookies, or API keys—credentials that can be issued, rotated, and revoked through established identity management practices. When an LLM processes requests, the model has effectively been granted implicit trust to interpret user intent and determine appropriate responses. An attacker who can influence what the model "understands" from an input has found a way to bypass application-level security controls that assume the input will be processed literally rather than interpreted contextually.

Second, the output space expands dramatically beyond what validation logic can address. Traditional input validation works because the expected input format is known: an email address follows a specific pattern, a quantity field accepts only integers within a defined range, a command must match a predefined set of options. When the input is natural language that the model must interpret, validation cannot enumerate all valid interpretations. A request to "show me my recent orders" and a request to "show me all orders from the database including those belonging to other users" might use similar vocabulary, yet one should be authorized and the other should not. The model must distinguish between these interpretations, but the security infrastructure has no mechanism to verify that the model's interpretation matches the developer's intent.

Third, the attack surface extends across the entire machine learning pipeline in ways that traditional security perimeters do not cover. The training data that shapes model behavior represents a potential attack vector. The model weights themselves are assets that might be compromised. The inference pipeline—how inputs are preprocessed, how the model is queried, how outputs are parsed—introduces additional interaction points where security controls must operate. Downstream integrations that consume model outputs assume those outputs are safe, but this assumption may be violated when attacks on the model propagate through the architecture.

The Interpretability Gap as Attack Surface

The interpretability gap refers to the structural uncertainty that exists between model input and model output—neither developers nor security tools can fully predict, explain, or verify how the model will respond to any given input. This gap exists not because of implementation bugs or missing features, but because of fundamental properties of how neural networks represent and process information.

To understand the security implications, consider how interpretability gaps affect different stakeholders. Developers building LLM-integrated applications face uncertainty about model behavior in production. The model may handle inputs that were never anticipated during development, may produce outputs that are technically correct but contextually inappropriate, or may fail in ways that reveal sensitive information about its training data or internal state. Security teams cannot apply traditional vulnerability assessment techniques because there is no code to audit, no binary to analyze, no protocol specification to verify. Penetration testing becomes fundamentally different when the target is a model whose behavior is learned rather than programmed.

The interpretability gap also affects how attacks succeed. Traditional attacks on software often exploit specific implementation flaws: a buffer that can be overflown, a query that can be injected, an authentication check that can be bypassed. These attacks work because the attacker's payload interacts with known code paths in predictable ways. AI attacks exploit the model's interpretive function rather than implementation flaws. A prompt injection attack succeeds not because of a bug in the implementation but because the model will interpret attacker-controlled text as instructions to be followed. The attack surface is the model's fundamental capability—the ability to understand and respond to natural language—which cannot be removed without defeating the purpose of deployment.

This creates a paradoxical security situation: the more capable the model, the larger its