The Evolving AI Security Landscape: Threat Models and Risk Frameworks

The Evolving AI Security Landscape: Threat Models and Risk Frameworks

The security paradigm that has governed software engineering for decades rests on a fundamental assumption: systems behave according to their specifications. A properly implemented authentication mechanism denies access to unauthorized users; an encrypted database protects its contents from eavesdroppers; an input validation routine prevents malformed data from causing buffer overflows. These expectations, while never absolute, provide a stable foundation for threat modeling and defensive planning. AI systems shatter this foundation in ways that security practitioners are only beginning to understand.

When a large language model produces output, no formal specification guarantees that the response will follow any particular logic, respect any particular boundary, or honor any particular constraint. The model's behavior emerges from statistical patterns in training data and the complex interaction between the input prompt and billions of parameters. This probabilistic, emergent nature—precisely what makes AI systems powerful—also makes them fundamentally different security targets. An attacker who has spent years learning to exploit SQL injection vulnerabilities may find that their expertise translates poorly to a system where the "code" they must manipulate consists of natural language prompts and where the "vulnerabilities" they must find exist in the probabilistic mapping between inputs and outputs.

This chapter addresses the foundational question that animates this entire volume: how do we systematically conceptualize and categorize AI security threats in ways that enable effective defense planning? The answer requires us to first understand what makes AI systems uniquely vulnerable, then to adapt traditional threat modeling frameworks for this new reality, then to learn from the real-world incidents that have already demonstrated the business impact of AI security failures, and finally to construct practical frameworks for prioritizing our defensive efforts. By the end of this chapter, you will have a complete threat model document template and risk assessment worksheet that you can adapt for your organization's specific AI deployments.

The Unique Attack Surface of AI Systems

Understanding AI security requires abandoning several comfortable assumptions that have guided traditional cybersecurity. These assumptions include the belief that system behavior is deterministic, that boundaries between system components are well-defined, that inputs follow predictable formats, and that outputs can be validated against explicit specifications. AI systems challenge each of these assumptions in ways that create novel attack surfaces.

Probabilistic Behavior as Attack Surface

Traditional software security often implicitly assumes that if we can identify the correct execution path through a program and ensure that an attacker cannot manipulate the program counter or alter memory contents, the system will behave as intended. This assumption fails catastrophically for AI systems because the relationship between input and output is fundamentally probabilistic rather than deterministic.

Consider a sentiment analysis model that classifies customer reviews as positive, negative, or neutral. A security analysis of traditional software would focus on whether the classification function correctly implements its specification, whether input validation prevents malformed data from causing crashes, and whether authentication mechanisms prevent unauthorized access to the classification service. For the AI system, we must ask additional questions: Can an attacker craft inputs that cause the model to misclassify with predictable patterns? Can adversarial examples cause the model to consistently misclassify inputs that appear identical to human observers? Can the model's confidence scores be manipulated to mislead downstream decision-making processes?

These questions have no clean answers because the model's behavior cannot be reduced to a finite set of execution paths. The same prompt run twice might produce subtly different outputs, and the security implications of these differences may be profound. An AI assistant that helps draft financial reports might occasionally include hallucinated statistics; an attacker who can increase the frequency or plausibility of these hallucinations has found a meaningful attack vector even without causing any crash or explicit malfunction.

The probabilistic nature of AI outputs also complicates the definition of "correct" behavior. In traditional software, a function either produces the correct output or it does not. For a large language model, "correct" behavior is often a matter of degree rather than kind. Is a response that captures the essence of what the user requested but includes one factual inaccuracy "correct"? Is a creative writing assistant that generates content slightly outside the style parameters it was given "malfunctioning" or simply showing appropriate creativity? These ambiguities create space for attacks that exploit the gap between formal specifications and human expectations.

External Data Dependencies and the Supply Chain Problem

AI systems derive their capabilities from training data, and this dependence creates attack surfaces that have no direct analogue in traditional software security. A conventional application receives data at runtime that might be malicious, but the application's core logic remains fixed. An AI system's "logic"—the patterns that determine its outputs—is encoded in model parameters that were determined during training. If an attacker can influence what data the model trains on, they can influence how the model behaves for all future users.

Training data poisoning represents one of the most insidious categories of AI attacks precisely because it operates during the system's development phase, when security oversight is often minimal. An organization that fine-tunes a language model on scraped web data, customer support transcripts, or employee-generated content has introduced numerous potential points of compromise. A malicious employee who can influence the composition of fine-tuning data might implant backdoors—specific trigger patterns that cause the model to exhibit anomalous behavior when activated.

The supply chain dimension of this problem extends beyond individual organizations. Many organizations use foundation models trained by third parties, incorporating them into products without full visibility into the training pipeline. The SolarWinds attack demonstrated the devastating potential of supply chain compromises in traditional software; analogous attacks on AI systems could be even more difficult to detect because the model behavior changes may be subtle and context-dependent.

External data dependencies also manifest at inference time. Retrieval-augmented generation systems that fetch relevant documents to inform model responses create dependencies on external data sources that may be untrusted or actively adversarial. A prompt injection attack against a retrieval system could cause the AI to incorporate malicious content into its context, influencing downstream outputs without directly targeting the model itself.

Complex Interaction Patterns and the Human-AI Interface

AI systems introduce interaction patterns that differ fundamentally from traditional software interfaces. In conventional applications, user input is processed through well-defined input validation layers before affecting system behavior. The security boundary between user and system is relatively clear. AI systems, particularly those powered by large language models, often accept natural language input that can be crafted in infinitely many ways, processed through model internals that remain opaque, and transformed into outputs that may have security implications far removed from the original input.

This complexity