Foundations of LLM Security: A New Attack Surface

Chapter 1: Foundations of LLM Security — A New Attack Surface

The question that animates this book is deceptively simple: *why does deploying a Large Language Model into a production application create security problems that do not map cleanly onto anything we already know how to defend?* Every enterprise that has shipped LLM features in the last three years has collided with some version of this confusion. Engineers with years of experience hardening web applications, microservices, and APIs find themselves unable to answer questions that used to be routine. Where is the input boundary? What counts as a "user"? Which data sources are trusted? Why does the same prompt produce different control-flow outcomes on different invocations? Why do defenses that worked against SQL injection, command injection, and cross-site scripting seem to do nothing against a paragraph of plain English?

The reason is that LLMs do not merely add a new component to the existing software stack. They introduce a fundamentally different kind of compute node — one that does not parse its inputs according to a fixed grammar, does not separate code from data, and does not maintain the clean process boundaries that classical security models assume. The rest of this chapter builds the conceptual scaffolding required to reason about these systems as security engineers. We will work through five arguments, each of which dismantles a specific assumption inherited from traditional application security, and then we will apply those arguments to a concrete system: a customer-support copilot connected to a knowledge base, a CRM, and a ticketing system. By the end of the chapter you should have a working vocabulary, a reference taxonomy (the OWASP Top 10 for LLM Applications), and a partially completed threat model that subsequent chapters will progressively harden.

1.1 From Deterministic Parsing to Probabilistic Interpretation

Traditional input validation rests on a contract between the application and the format of its inputs. That contract is embodied in a *parser* — a deterministic procedure that either succeeds and produces a structured object, or fails and produces an error. Whether the parser is a regular expression enforcing an email shape, an XML schema validating a SOAP envelope, or a SQL tokenizer building a parse tree from a query string, the security-relevant property is the same: there is a sharp, mathematically defined boundary between *valid input* and *invalid input*. Every downstream security control — parameterized queries, output encoding, prepared statements, allow-lists, length caps — assumes the existence of that boundary. Remove it, and the entire chain of reasoning collapses.

LLM-driven systems do not have a parser in this sense. They have a *decoder*: a procedure that, given a sequence of tokens representing the conversation so far, samples the next token from a probability distribution, appends it, and repeats. The model's input is not validated; it is *consumed*. The model does not return a parse error when a user submits something unexpected — it produces a continuation. That continuation is constrained by training, by alignment, by the system prompt, and by the temperature hyperparameter, but it is not bounded by a grammar. The boundary between "this is a well-formed request" and "this is malformed or hostile" does not exist as a discrete object inside the model. It is approximated, statistically, across the joint distribution of all token sequences the model could produce.

This has three immediate consequences for security engineering.

First, every token in the input is simultaneously data and instruction. In a SQL parser, the string `'; DROP TABLE users;--` is unambiguously a chunk of data — characters that the tokenizer walks through, the parser grammar rejects, and the database never executes. In an LLM, the same string is not parsed; it is embedded into a high-dimensional space and attended to. The model has no syntactic category called "instruction" that can be lexically isolated from "data." Anything that influences the next-token distribution — a retrieved document, a tool result, a chat history line, a system prompt — is functionally equivalent to the user's prompt for the purpose of producing the next token. This is what researchers mean when they say LLMs suffer from a *lack of instruction–data separation*. It is not a bug in any one model; it is a structural property of decoder-only transformers trained on next-token prediction.

Second, the output of the model is not a structured object with a schema. It is a sequence of tokens whose shape depends on the model's learned prior over what continuations are likely, conditioned on the prompt. When we ask an LLM to "extract the customer's account number from this email and return it as JSON," we are not invoking a JSON parser — we are asking the model to behave as if it were one. Sometimes the model obliges. Sometimes it produces prose, or partial JSON, or JSON with plausible-but-wrong fields, or JSON embedded inside a Markdown code fence. Every downstream consumer that assumes the output *is* JSON in the way a parser-produced JSON object is JSON is making an unfounded assumption. The model is a *generator* of text that *looks like* a structured response, not a producer of structured responses.

Third, the same prompt can produce materially different control-flow consequences. Set `temperature=0.2` and a benign-looking prompt about "summarizing the most recent customer emails" might cause the model to call a `list_emails` tool; set `temperature=0.9` and the same prompt might cause the model to skip the tool call and just say "I don't have access to recent emails." From a security perspective, this means that *probabilistic properties* of the system — not just deterministic inputs — shape the attack surface. An adversary who can drive the model into a high-entropy region of its output distribution can make defenses that rely on the model "usually" doing the right thing unreliable in exactly the conditions that matter.

A worked example clarifies the gap. Consider a small piece of code that, today, would be considered industry-standard safe input handling:

```python def lookup_customer(customer_id: str) -> dict: if not customer_id.isalnum() or len(customer_id) > 32: raise ValueError("invalid customer id") cursor.execute( "SELECT name, email FROM customers WHERE id = %s", (customer_id,) ) return cursor.fetchone() ```

The validation is correct. The parameter binding is correct. The SQL injection vector is closed. If we naively translate this into an LLM-driven version, we might write:

```python def lookup_customer(customer_id: str) -> dict: if not customer_id.isalnum() or len(customer_id) > 32: raise ValueError("invalid customer id") prompt = ( f"Look up customer {customer_id} in the CRM and return " f"their name and email as JSON." ) response = llm.complete(prompt, tools=[crm_lookup_tool]) return json.loads(response.text) ```

The first four lines still work as intended: malformed IDs are rejected before the LLM is ever called. But the security boundary at the LLM is not the SQL injection boundary. It is something like: *the model must (a) decide to call the CRM lookup tool, (b) call it only with the validated `customer_id`, (c) return only the requested fields, and (d) return them as parseable JSON that reflects exactly what the tool returned.* None of these four conditions is enforced by code. They are *induced* by the prompt, *constrained* by the model's training, and *verified* (if at all) by downstream parsing. Each is a probabilistic promise, not a structural guarantee.

The implication is not that input validation is useless in LLM systems. Input validation against the *external* client — rate limits, length caps, content-type checks, authentication — remains essential. What changes is th