Introduction to AI Security Engineering: Why Defending Models Matters

Chapter 1: Introduction to AI Security Engineering: Why Defending Models Matters

Why AI Systems Need Their Own Security Discipline

The first time a major airline was tricked into issuing a refund based on instructions a customer pasted into a chatbot, the security community was forced to confront an uncomfortable reality: traditional defenses do not stop attacks that arrive in natural language. The customer did not exploit a buffer overflow, did not phish an employee, and did not bypass a firewall. They typed a polite paragraph into a chat window. The system complied. Within hours, screenshots of the interaction spread across social media, and the company's legal and engineering teams were left explaining how a language model had unilaterally committed a financial action it had no business authorizing.

That incident, and dozens like it reported in 2023 and 2024, expose a structural gap in how organizations think about security. Most enterprises spent two decades hardening networks, patching operating systems, training employees to spot phishing emails, and segmenting sensitive databases. Those investments matter. They are not wrong. But they assume an attacker will interact with software through well-defined interfaces: an API call with typed parameters, a SQL query against a known schema, a request signed by an authenticated user. A large language model (LLM) attached to a company's data and tools does not present those kinds of interfaces. It presents a conversation. Anything that can be expressed in natural language becomes a potential command. Anything the model can read becomes a potential payload. Anything the model can do becomes a potential unauthorized action. The result is a system whose attack surface is, in a meaningful sense, the entire English language, plus every document the model can retrieve, every function it can call, and every user who can speak to it.

This is why AI security engineering must be treated as a distinct discipline rather than a sub-topic of cybersecurity. The two fields share a common ancestor in risk management, and many of the same principles apply. But the threat actors, the attack surfaces, the failure modes, and the defensive techniques are different enough that an organization relying solely on traditional security expertise will systematically miss the most important risks. The OWASP Top 10 for LLM Applications, published in 2023 and updated in 2025, has become the de facto reference for the field, and the threats it documents — prompt injection, sensitive information disclosure, supply chain vulnerabilities, data poisoning, model theft, and others — are not just old vulnerabilities in new clothes. They emerge from properties unique to statistical language models: their inability to distinguish instructions from data, their tendency to follow patterns even when those patterns contradict prior training, their dependence on retrieved context, and their willingness to be steered by social pressure expressed in text.

Let us be precise about what AI security engineering is. At its core, it is the practice of designing, building, deploying, and operating AI systems — particularly systems that incorporate LLMs — in a way that preserves their intended behavior in the face of adversarial inputs, malicious actors, accidental misuse, and the long tail of edge cases that statistical models inevitably produce. It encompasses threat modeling specific to AI architectures, defensive design patterns such as input filtering and output validation, testing methodologies like red-teaming and adversarial evaluation, monitoring strategies for runtime detection of attacks, and governance processes for tracking model versions, training data, and access controls. It also includes the often-overlooked discipline of AI safety in the sense of preventing models from producing harmful, biased, or misleading outputs, because from a security perspective, a model that can be trivially manipulated into generating disallowed content is just as compromised as one that leaks a database.

How is this different from traditional cybersecurity? Several ways are worth understanding deeply.

First, the trust boundary is blurred. In a conventional web application, the server trusts only authenticated requests and validates the structure of every input against a strict schema. A request that does not match the schema is rejected before it ever touches business logic. An LLM application, by contrast, often treats natural language as both the interface and the data. There is no schema to validate a sentence against. The model is expected to handle arbitrary, unpredictable, and potentially adversarial text, and it must do so while still producing outputs that are useful. This means the input validation step, which is the cornerstone of traditional application security, becomes radically harder. You cannot reject a prompt just because it is unexpected; an LLM application is precisely designed to handle the unexpected. Yet the same openness that makes the system useful also makes it vulnerable.

Second, the system's behavior is non-deterministic and data-dependent. Traditional software does what its code says, every time. A model does what its weights encode, conditioned on the prompt it receives, which means the same input can yield different outputs in different contexts, and a slight rewording of a prompt can change the model's decision. This makes reasoning about security properties — "this input will never cause the system to do X" — fundamentally harder. A secure-by-design web application might be provably immune to a class of attacks. A secure-by-design LLM application is, at best, probabilistically resistant, and the probability is a function of training data, fine-tuning, prompt engineering, retrieval content, and tool configurations that can all change after deployment.

Third, the attack surface is partly determined by capabilities the model is given. A language model with no tools and no data access is comparatively boring from an attacker's perspective. The moment you connect it to a database, an email system, a code execution environment, or a payment API, you have given it powers, and an attacker who can manipulate the model can exercise those powers. This means the security of an AI system is not just a property of the model; it is a property of the entire stack around it. Securing the model without securing the integration is like installing a high-security front door on a house with no walls.

Fourth, the threat actors have new capabilities and new incentives. Nation-state actors, criminal organizations, and individual hackers have all discovered that LLMs lower the cost of crafting convincing phishing, generating malware, and scaling social engineering. Insider threats now include the possibility of an employee who fine-tunes a model on sensitive data, who disables a safety filter, or who grants the model access to a system it should not touch. The supply chain for AI includes pre-trained models downloaded from public repositories, training datasets scraped from the open web, and third-party APIs that may themselves be compromised. None of these risks are entirely new, but their combination in AI systems creates a threat landscape that traditional security frameworks were not designed to navigate.

Given all of this, you might reasonably ask: is AI security engineering a passing trend, a rebranding of old problems, or a genuinely new field? The honest answer is that it is genuinely new in its particulars while drawing deeply on established fields. Cryptography taught us how to reason about adversarial inputs; AI security applies that mindset to natural language. Software engineering taught us about input validation, least privilege, and defense in depth; AI security reinterprets those principles for systems whose inputs and outputs are not structured. Risk management taught us to prioritize based on likelihood an