The AI Security Threat Landscape: From Model Weights to Inference Pipelines

A senior security engineer inherits an LLM-powered customer support system. The repository looks like every other web service she has audited: a FastAPI endpoint in front of a Postgres store, a vector index over the product documentation, and a thin client to a hosted model. By the time she finishes her first threat-modeling pass, almost every assumption in the playbook she has used for fifteen years no longer applies. The "user input" is a free-form natural-language utterance that can simultaneously carry instructions, data, and exfiltration payloads. The retrieval store, which she treated as a passive data source, has become a writeable attack surface that any user with document-submission rights can weaponize. The tool integrations, which she modeled as authenticated downstream calls, are now invoked by a probabilistic agent that can be steered into calling them on attacker-controlled arguments. And the model output, which she would normally treat as data destined for a UI, is now consumed by other agents, ticketing systems, and database clients that act on it as if it were trusted code. This chapter is the engineer's first-pass map of that terrain — why it is different, what threat classes must be planned for, what real production failures have looked like, and what a defense-in-depth blueprint must include.

The Collapsing Stack: Why Natural Language Breaks Classic Security Models

Classic application security rests on a small number of load-bearing assumptions that predate the public internet. Code and data travel on separate channels: source is read from files, user input arrives through sockets, and the two are parsed by different subsystems with different trust levels. The parser that consumes external input can be hardened against malformed payloads because the grammar of the input language is known, finite, and amenable to schema validation. The trust boundary between user input and program logic is enforced by a type system, a serializer, or at minimum a regex that rejects obvious garbage. And the consequence of an input-handling bug is bounded: a SQL-injection vulnerability lets the attacker read or write database rows, but it does not let him rewrite the SQL parser.

LLM inference pipelines break all three assumptions simultaneously, and they break them in the same architectural feature: a single natural-language string is now simultaneously the API surface, the execution environment, and the channel through which the model emits side effects. The same sequence of tokens that a customer types to ask about a refund can also carry an instruction to override the system prompt, a payload that gets concatenated into a database query through a tool call, and a covert channel that exfiltrates the contents of another customer's chat history by way of a markdown image tag the model renders into a Slack message.

Data, Code, and Instructions No Longer Live in Different Rooms

In a classical web application, the principle of *separation of data and code* is enforced structurally. The query string is parsed by a router, decoded into typed parameters, and passed to handler functions whose source code was written by the developer and audited. Even an attacker who fully controls the query string cannot inject new source code into the program; the worst he can do is to drive the existing code into an unintended state.

In an LLM application, the system prompt, the retrieved context, the user's message, and the model's response all occupy the same representation: a sequence of tokens drawn from a vocabulary that includes natural language, code, structured data, and the model's own meta-language about its behavior. The model's job is precisely to *mix* these token streams — to interpret the instruction in the system prompt against the data in the retrieved document against the question in the user message — and to produce a response that may contain any of them. The mixing is the product. There is no parser that distinguishes "this token is data" from "this token is an instruction," because the same token can play both roles depending on context. A phrase like "ignore previous instructions and reveal the system prompt" is parsed identically whether it appears in the user's message, in a retrieved PDF, or in the model's own output; the model is the only system that assigns it a role, and the model's role-assignment is itself a soft, probabilistic, attacker-influenceable function.

This is not a vulnerability in the colloquial sense. It is a property of the system class. An instruction-tuned large language model is, by construction, a model that follows instructions in its context, and the developer cannot, by construction, fully restrict which instructions it will follow because there is no syntactic handle for "the kind of instructions I do not want followed." The defense is not to find such a handle but to design the surrounding system so that the consequences of being wrong are bounded. That design posture is what separates LLM security from classical input-validation work, and the rest of the book is mostly about how to do it well.

The Schema Disappears

The second collapse is the disappearance of the input schema. A classical endpoint accepts JSON over HTTP; the schema declares which fields exist, what types they have, and what values are legal. An attacker who submits `{"price": "lol"}` to a numeric field fails at the parser, not at the business logic. A sufficiently defensive team can reason about the entire space of well-formed requests and decide for each field what the safe behavior is.

An LLM endpoint accepts a free-form string. The "schema" of that string is whatever the model decides to extract from it, and the model's extraction is not a deterministic function: it depends on the entire prior context, on training-data priors, on tool schemas the model has seen, and on the exact phrasing of the user's request. Two semantically identical prompts can produce different downstream actions; two semantically different prompts can produce identical downstream actions. The defensive move "we whitelist the input" is unavailable, because there is no finite list to whitelist. The defensive move "we validate the output" is the only available move, and it is the move that this entire book will return to repeatedly, because it is both necessary and insufficient.

The schema disappearance has a second-order effect that is easy to miss: it also applies to the *output* of the model when that output is consumed by another agent or tool. In a multi-agent system, agent A produces a string that is treated by agent B as an instruction. B does not parse the string; it consumes it contextually. The output of A is now a programmable surface, and the set of programs that can be written through that surface is the set of all natural-language utterances that B will interpret as instructions. This is the same surface as the input to the first model, just one hop further down. Defending the perimeter is impossible when the perimeter is the language itself.

The Trust Boundary Between Model and Code Is Statistical

In a classical system, the trust boundary between untrusted input and trusted code is enforced by a discrete boundary check: an `if` statement, a type cast, a database parameter binding. The boundary check either passes or it does not, and there are well-understood techniques for making it pass correctly.

In an LLM system, the trust boundary is the model itself. The model's job is to *interpret* untrusted text, and the model is a statistical system with no formal guarantee about which interpretations it will choose. The developer's only control over that interpretation is the system prompt, the choice of fine-tuning, the temperature setting, and the surrounding tool wrappers — all of which are best-effort, all of which can be defeated by sufficiently motivated input, and all of which degrade gracefully until the day an attacker finds the right