Foundations of LLM Security Engineering

Foundations of LLM Security Engineering

A large language model integrated into an application is not a software component in any sense a classical security engineer recognizes. It does not parse a typed interface, it does not validate its own inputs, and it does not honor a syntactic boundary between the instructions it should execute and the data it should merely read. When text enters its context window, the model does not know—and cannot know—whether that text is a directive from the system operator, a query from the user, or untrusted content fetched from an arbitrary URL. Every assumption that classical input handling rests upon fails inside the transformer. This chapter establishes the conceptual vocabulary for reasoning about that failure: the trust boundaries that no longer hold, the threat surfaces that classical models do not predict, the architectural patterns that determine how exposed a given system is, and the engineering posture—treating the model as a semi-trusted computing substrate—that organizes every defense discussed in subsequent chapters.

1.1 The Instruction–Data Confusion Problem and the Absence of Syntactic Input Trust Boundaries

Every classical computing system that consumes user input solves the same underlying problem: distinguishing what the system should *do* from what it should merely *read*. SQL parameterization separates the query structure from the values supplied to it. HTML escaping prevents user-supplied content from being interpreted as markup. Template engines introduce sandboxed substitution contexts. Operating systems separate executable memory from data memory with NX bits. In every case, the security guarantee rests on a syntactic property of the input channel: there is a structural marker—an operator, a delimiter, a type tag—that the interpreter can use to decide whether a given token is instruction or data.

The large language model is the first widely deployed computational substrate in which this guarantee does not exist. The model's input is a sequence of tokens drawn from a vocabulary; tokens may originate from the system prompt, the conversation history, retrieved documents, tool results, or the user's latest message, and once concatenated into the context window, they are syntactically indistinguishable. There is no equivalent of a parameter binding, no quote-escape discipline, no NX bit. The model is trained to treat all natural-language content as potentially directive, because the training objective itself is to follow instructions expressed in natural language. This is the *instruction–data confusion problem*, and it is the single architectural fact from which almost every other LLM security issue derives.

The problem is not new in computer science. SQL injection, cross-site scripting, command injection, email header injection, and template injection are all instances of an interpreter confusing data for code. What is new—and what makes LLM injection qualitatively different—is the absence of a complete mitigation. SQL injection can be eliminated by using prepared statements because the underlying interpreter honors a syntactic distinction between statement structure and bound parameters. HTML injection can be eliminated by proper escaping because HTML parsers honor a syntactic distinction between text and tag content. There is no analogous mitigation for LLMs because the model is explicitly trained to *ignore* such distinctions when they would prevent it from following user intent. A delimiter that the model cannot be tricked into crossing is, by definition, a delimiter the model cannot use to receive instructions. Robustness and utility are in tension at the level of the architecture itself.

This tension has several immediate consequences for engineering practice. First, any byte of text that the model reads is a potential control channel. If a system retrieves web pages, the contents of those pages may include adversarial instructions. If the system reads email, the body of an incoming message may contain directives. If the system inspects file uploads, the contents of those files may contain directives. The implication is that the entire retrieved context must be treated as untrusted input, regardless of how the data was originally classified in the system's data model. A document marked "internal" in the corporate content management system is, from the model's perspective, an arbitrary sequence of tokens that may carry instructions.

Second, there is no input validation that meaningfully reduces this risk in the way that input validation reduces classical injection risk. Length limits, character filters, and signature-based scanners can remove obvious attack patterns, but the attack surface is the entire space of natural-language phrasings that successfully redirect model behavior. New phrasings are discovered continuously; no maintainable allowlist covers them. Defense must instead be relocated to where control is exercised—principally at the boundary between the model's outputs and any side-effecting action they trigger.

Third, the problem is recursive across compositional systems. A model that invokes a tool

A model that invokes a tool produces a piece of text—typically a structured blob naming a function and its arguments—which is dispatched to an execution layer outside the model. The execution layer returns text (often a JSON document, an error string, or the raw contents of a fetched resource), and that text is concatenated into the next context window. If any tool in this chain retrieves content controlled by an adversary—a webpage, a search result, a database row written by a customer, an email body, a document pulled from a shared drive—that adversary's text has been admitted to the model in the same syntactic position as the system prompt. The two are, token-wise, indistinguishable. The model will receive them as instructions, as data, or as some inextricable mixture, depending on phrasings its training has made salient. This is *indirect prompt injection*, and it has become the dominant attack pattern in deployed LLM systems precisely because the adversary does not need to compromise any component the operator owns; they need only to publish content the system will eventually read.

The term deserves care. *Direct* prompt injection is the situation in which the adversary is the user—the party typing into the chat box. From the system operator's perspective this is at least an authorized channel; the user is, in some sense, supposed to be issuing instructions. Indirect prompt injection is the situation in which the adversary is a third party whose text reaches the model through some other path: a web fetch, a retrieved document, a tool result, a multi-tenant database record, a calendar invite, a Slack message, a PDF attached to a customer-support ticket. The text is "data" by every classical classification, but the model cannot honor that classification.

Classical injection defenses assume the interceptor—the SQL parser, the HTML rendering engine, the shell—will reliably distinguish instructions from values. The defense lives inside the interpreter. For LLMs, the defense must live elsewhere. Three locations are available: *before* the text enters the model (filtering and sanitization, which is unreliable), *inside* the model's reasoning (instructional hardening, which the attacker paraphrases around), and *after* the model emits its response (constrained execution, which is the load-bearing wall). The remainder of this chapter explains why the third location is the only one on which a principled defense can be built, and what a defense layered there has to look like.

1.2 Trust Boundaries in LLM-Mediated Systems

A *trust boundary* is the locus at which authority changes hands. In a classical system the boundary is sharp and inspectable: the network perimeter is crossed by packets any firewall can examine; the SQL boundary is crossed by strings any prepared statem