AI Security Foundations and Threat Modeling | AI安全基础与威胁建模

AI Security Foundations and Threat Modeling | AI安全基础与威胁建模

> *"The security of an AI system cannot be treated as an afterthought—it must be architected into every layer of the machine learning pipeline, from data collection to model deployment."* > > *"AI系统的安全性不能被视为事后考虑，而必须被架构到机器学习管道的每个层面，从数据收集到模型部署。"*

The fundamental challenge confronting security practitioners today is not merely defending static systems against known attack vectors, but modeling threats against artifacts that are inherently stochastic, continuously evolving, and capable of exhibiting behaviors that emerge unpredictably from their training data. Traditional threat modeling methodologies—developed for deterministic software systems where outputs can be precisely predicted from inputs—break down when confronted with machine learning models that may produce wildly different outputs for semantically similar inputs, learn behaviors that were never explicitly programmed, or fail in ways that cannot be reproduced under identical conditions. This chapter establishes the conceptual foundation for systematically understanding, categorizing, and mitigating security threats in AI systems.

本章要解决的核心问题是如何建立一套系统性的方法来理解和建模AI系统面临的安全威胁。这不仅要求我们理解传统软件安全与AI安全的本质差异，更需要我们掌握适用于AI生命周期的威胁建模方法论。在本章中，我们将从AI安全的独特属性出发，深入分析AI系统全生命周期的攻击面，探讨STRIDE、PASTA等经典框架在AI场景下的适配与扩展，引入ATT&CK for AI作为结构化的威胁分析工具，并最终通过一个实践案例展示如何将理论框架转化为可操作的威胁模型。

The Essential Distinction: AI Security vs. Traditional Software Security | 本质区别：AI安全与传统软件安全

Understanding why AI security demands fundamentally different approaches requires first grasping the ontological differences between traditional software and machine learning systems. Traditional software is built on explicit logic: programmers write code that defines exact rules for how inputs should be transformed into outputs. The system behaves precisely as specified; unexpected behavior indicates a bug that can be diagnosed by tracing through the code path. Security in this paradigm focuses on ensuring that the logic is correct, the implementation matches the specification, and attackers cannot manipulate inputs to trigger unintended code paths.

理解AI安全为何需要根本不同的方法，首先需要理解传统软件与机器学习系统之间的本体论差异。传统软件建立在显式逻辑之上：程序员编写代码，定义输入如何转换为输出的精确规则。系统按照规范精确运行；意外行为表明存在可以通过追踪代码路径来诊断的缺陷。该范式下的安全性关注于确保逻辑正确、实现符合规范、以及攻击者无法操纵输入以触发意外的代码路径。

Machine learning systems, by contrast, learn their behavior from data rather than explicit programming. A neural network does not contain handcrafted decision rules; instead, it discovers statistical patterns in training data and encodes these patterns in millions or billions of numerical parameters. The "code" of a neural network is not human-readable logic but a high-dimensional parameter space whose structure emerged from optimization processes. This fundamental shift in how systems acquire their behavior creates security challenges that do not exist—or are far less severe—in traditional software.

相比之下，机器学习系统从数据中学习行为，而非通过显式编程。一个神经网络不包含手工制作的决策规则；相反，它从训练数据中发现统计模式，并将这些模式编码在数百万或数十亿个数值参数中。神经网络的"代码"不是人类可读的逻辑，而是一个高维参数空间，其结构源于优化过程。这种获取行为方式的根本转变创造了传统软件中不存在的安全挑战——或者在该范式下要轻微得多。

Randomness, Uncertainty, and Emergent Behavior | 随机性、不确定性与涌现行为

The first dimension of this distinction lies in the inherent randomness embedded in AI systems. Even when an AI system is not explicitly designed to be probabilistic, its behavior often exhibits stochastic characteristics due to several factors. Modern neural networks contain dropout layers, batch normalization, and other components that introduce randomness during inference. Many AI applications intentionally incorporate randomness—for example, large language models sample their outputs from probability distributions over tokens, meaning that the same prompt may elicit substantively different responses on different invocations. This non-determinism means that security testing cannot rely on reproducing identical outputs from identical inputs; a security control that blocks a specific output today may fail to block a semantically equivalent output generated through a different sampling path tomorrow.

这种区别的第一个维度在于AI系统中嵌入的固有随机性。即使AI系统没有明确设计为概率性的，其行为通常也表现出随机特征，这源于多种因素。现代神经网络包含dropout层、batch normalization等组件，这些在推理过程中引入随机性。许多AI应用有意结合随机性——例如，大语言模型从词元概率分布中采样输出，这意味着相同的提示在不同调用时可能产生实质上不同的响应。这种非确定性意味着安全测试不能依赖于从相同输入重现相同输出；今天阻止特定输出的安全控制，明天可能无法阻止通过不同采样路径生成的语义等效输出。

Uncertainty pervades AI systems at multiple levels. Epistemic uncertainty—uncertainty about the model itself—arises because we rarely know whether our training data adequately represents the distribution of inputs the model will encounter in deployment. A model trained on data from one geographic region may fail catastrophically when deployed in another, not because of an explicit attack but because the model has learned spurious correlations that happen to hold in the training distribution. Aleatoric uncertainty—intrinsic unpredictability—arises from the fundamental indeterminacy of many real-world phenomena. A weather prediction model cannot provide a single correct temperature forecast because weather is genuinely chaotic; the best it can offer is a probability distribution over possible outcomes. Security systems built on such models must account for this fundamental unpredictability rather than assuming crisp, deterministic boundaries between safe and unsafe outputs.

不确定性在多个层面渗透AI系统。认知不确定性——关于模型本身的不确定性——源于我们很少知道训练数据是否充分代表了模型在部署中将遇到的输入分布。在一个地区数据上训练的模型在另一个地区部署时可能灾难性地失败，这不是因为明确的攻击，而是因为模型学习了恰好在训练分布中成立的虚假相关性。偶然不确定性——固有不可预测性——源于许多现实世界现象的基本不确定性。天气预测模型无法提供单一正确的温度预测，因为天气本质上是混沌的；它能提供的最好结果是可能结果的概率分布。基于此类模型构建的安全系统必须考虑这种基本不可预测性，而非假设安全与不安全输出之间存在清晰的确定性边界。

Most alarmingly, AI systems can exhibit emergent behaviors—capabilities and vulnerabilities that were not anticipated by their designers and cannot be easily predicted from analyzing individual components. Large language models have demonstrated abilities to perform arithmetic, translate between languages, write code, and engage in multi-step reasoning despite never being explicitly trained to do these things. These capabilities emerged from the training process in ways that remain only partially understood. Similarly, security vulnerabilities may emerge from the interaction of model capabilities with deployment context in ways that are difficult to foresee. A model that appears safe during development may exhibit harmful emergent behaviors when deployed at scale, when users interact with it in unanticipated ways, or when it encounters inputs that differ subtly from training distribution.

最令人担忧的是，AI系统可以表现出涌现行为——设计师未预料到的能力和漏洞，无法通过分析各个组件轻松预测。大语言模型展示了执行算术、在语言间翻译、编写代码和进行多步推理的能力，尽管从未被明确训练做这些事情。这些能力以仅部分被理解的方式从训练过程中涌现。同样，安全漏洞可能从模型能力与部署环境的交互中涌现，这些交互难以预见。在开发过程中看起来安全的模型，在大规模部署时、在用户以出乎意料的方式与其交互时、或在其遇到与训练分布略有不同的输入时，可能表现出有害的涌现行为。

The Attack Surface Expansion | 攻击面的扩展

Traditional software security focuses on protecting well-defined boundaries: the network perimeter, the application layer, the database. Attackers may exploit vulnerabilities in any of these components, but the components themselves are static artifacts—code that was written, reviewed, tested, and deployed as discrete units. AI systems fundamentally expand this attack surface because the "code" itself—the trained model—can be attacked, modified, or stolen at multiple points in its lifecycle.

传统软件安全专注于保护明确定义的边界：网络边界、应用层、数据库。攻击者可能利用这些组件中的任何漏洞，但组件本身是静态产物——作为离散单元编写、审查、测试和部署的代码。AI系统从根本上扩展了此攻击面，因为"代码"本身——训练好的模型——在其生命周期的多个阶段可能受到攻击、修改或窃取。

Consider the implications: a traditional software vulnerability exists in code that can be patched. An AI system vulnerability may exist in a model whose behavior is determined by training data and optimization processes. Patching a model to remove a learned vulnerability may be far more difficult than patching code—it may require identifying which training examples contributed to the vulnerability, retraining on a curated dataset, and verifying that the remediation did not introduce new vulnerabilities or degrade legitimate functionality. The attack surface of an A