Foundations of AI Security Engineering: Adversarial Models and the New Attack Surface

A security engineer inspecting an LLM-integrated production system in 2024 confronts an unfamiliar topology. Surrounding the conventional stack of load balancers, REST APIs, relational databases, and container orchestrators sit new components whose semantics resist classical analysis: a prompt template written in natural English prose, a function-calling layer that translates that prose into structured tool invocations, a vector store that ships retrieved documents into the same context window as system instructions, and a stochastic runtime that may emit a different answer to the same question on consecutive invocations. The defensive reasoning that protects a Java microservice—input validation at the boundary, parameterized queries, output encoding, least-privilege access control—does not vanish in this environment, but it is no longer sufficient on its own. A new class of vulnerability has emerged, one in which the "code" executed by the system is partially written in a natural language that any user can influence, and in which the boundary between trusted instruction and untrusted data has been deliberately collapsed. This chapter is the conceptual foundation for the engineering discipline that addresses this problem. It formalizes the adversary, redefines the trust boundary, distinguishes AI security from the related fields it inherits from, and introduces a reference architecture—the LLM System Stack—against which the remainder of the book will reason.

The Collapse of Classical Security Assumptions

Classical information security rests on a small set of load-bearing assumptions. The first is that program logic and data are categorically distinct: code is parsed by a deterministic interpreter, data is processed as opaque bytes, and the two are never confused. Buffer overflows, SQL injection, and command injection are the canonical violations of this assumption, and the defensive response is a well-trodden path of input validation, escaping, and parameterized interfaces. The second assumption is that the runtime is deterministic enough to permit reproducible testing. Given the same input and the same system state, a well-instrumented system should produce an equivalent output, allowing security tests to be re-run, regressions to be detected, and patches to be validated. The third assumption is that the trust boundary is well-defined: there is a clear perimeter at which untrusted input is sanitized before it reaches trusted logic, and the perimeter is enforced by code that itself is trusted.

Each of these assumptions is partially violated by an LLM-integrated system. The model receives a single concatenated string—the "context window"—in which system instructions, retrieved documents, conversation history, and user input are interleaved as natural language. The model is trained to treat all of this text as a single stream of instructions to be followed, because language modeling is fundamentally a task of "what would a helpful assistant say next given this entire context?" The "code" executed by the system is therefore not a separate syntactic object but a paragraph of English prose, and the "data" that the code operates on is other paragraphs of English prose, drawn from sources that range from fully trusted (developer-written templates) to fully untrusted (a scraped web page returned by retrieval). The classical separation between code and data is not merely blurred; it is dissolved, and along with it the defenses that depend on that separation.

The second assumption fails because the same prompt, when submitted twice to the same model with the same seed, may produce different outputs, particularly at non-zero sampling temperature. A security test that asserts "this prompt does not exfiltrate the system prompt" is not a stable assertion: it may pass on invocation $N$ and fail on invocation $N+1$. The third assumption fails because the trust boundary is no longer a single perimeter but a moving, semantic gradient: the system prompt at the top of the context window is trusted, the retrieved document is partially trusted, the user's latest message is untrusted, and the model must perform an implicit, semantic judgment about which parts of its input to obey. There is no parser that can mechanically distinguish instruction from data; the distinction must be inferred, and inference is exactly what an adversary can attack.

These failures are not transient artifacts of immature tooling. They are structural consequences of treating natural language as an instruction set. Any system that uses a foundation model as its reasoning substrate will inherit them, regardless of which model, which vendor, or which fine-tuning regime is used. The defensive response must therefore be structural, not incidental: a discipline-level rethinking of how adversarial reasoning is performed when the attack surface is the model's interpretation of a context window rather than the network reachability of a daemon.

The Adversarial Model: Direct, Indirect, and Insider Threats

A rigorous security analysis begins with an explicit adversary. Traditional threat modeling names the adversary in terms of position (network, host, insider), capability (unauthenticated, authenticated, privileged), and intent (opportunistic, targeted, nation-state). For LLM-integrated systems, the same template applies, but the categories acquire new content and a new class of adversary emerges.

Direct adversaries are users of the system who attempt to manipulate the model through the legitimate input channel. Their attack surface is the user-facing surface of the model: the chat box, the API endpoint that accepts prompts, the tool that injects user content into a templated query. A direct adversary cannot directly touch the system prompt, the model's weights, or the retrieved documents, but they can craft inputs designed to override the system prompt, exfiltrate it, cause the model to call tools in unintended ways, or produce output that violates a business rule. Prompt injection in its direct form—"ignore previous instructions and…"—is the archetype, but the surface is much broader: it includes multi-turn escalation, encoding-based evasion, role-play jailbreaks, and the exploitation of any tool or function that the model can invoke on the user's behalf. Direct adversaries are the easiest to reason about because their position is well-defined: they are at the user boundary, and any defense that operates at that boundary—input filters, output filters, model-level alignment—can be evaluated against them.

Indirect adversaries compromise content that the system ingests on the user's behalf but does not originate from the user. The principal vector is the retrieval-augmented generation (RAG) pipeline: the system fetches documents from a corpus—an internal knowledge base, a search engine, a customer-uploaded file, a public web page—and inserts them into the context window. An attacker who can place content into that corpus has injected instructions into the model's input without ever talking to the model. The attacker might be a user uploading a "resume" parsed by a recruitment assistant, a third party publishing a web page indexed by a browsing agent, a malicious document in a shared drive consumed by a summarization tool, or a poisoned embedding in a vector store. Indirect injection is structurally harder to defend than direct injection because the attacker's content is often outside the defender's control and is processed by the model in the same context as the system prompt. The semantic trust boundary that was supposed to keep the user outside the trusted zone is now bypassed by the very mechanism—retrieval—that was introduced to make the system more useful.

Insider threats occupy positions that traditional insider threat models recognize—a malicious or compromised employee, a contractor, a third-party developer—but their capabilities are extended in the LLM era. An insider can