AI安全工程的学科定位与威胁全景

第 1 章 AI安全工程的学科定位与威胁全景

When a probabilistic language model sits inside a deterministic business process, who is responsible when its output causes harm? The question is operational, not rhetorical. In 2023 it landed on the desks of AppSec teams that had no model for "validation" of an open-ended output. In 2024 it landed on ML Security researchers who realized that most consequential failures occur outside the model artifact. In 2025 it landed on AI Alignment researchers who had spent decades asking *whether the model wants to do the right thing* but had not built the discipline of *preventing it from doing the wrong thing*. None of these three groups owns the engineering of building the protection. That engineering is the subject of this book.

This chapter defends three claims:

1. AI Security Engineering (AI安全工程) is an independent engineering discipline, not a sub-field of ML Security or a future-state extension of Application Security. 2. Its object of study can be specified through an ontological quartet—Assets, Threats, Capabilities, Roots of Trust (资产-威胁-能力-信任根)—which serves as both vocabulary and diagnostic instrument. 3. Its maturity can be measured; most organizations in 2026 still operate at Level 0 or Level 1 of a five-level model that this chapter introduces.

The remainder of the book is organized around this quartet. The present chapter lays the foundation; Chapter 2 takes the first quadrant—threat modeling—and applies structured techniques to the seam-level attack surface of a deployed LLM application.

1.1 学科问题：被三条既有边界遗漏的工程地带

The argument for independence rests on a single observation: *the failure modes of AI-integrated systems do not live inside any of the three existing security disciplines*. They live at the seam where probabilistic generation meets deterministic execution.

Traditional Application Security (AppSec, 传统应用安全) is built on a deterministic input-output contract: given input $I$, function $f$ produces output $O = f(I)$, and any deviation is a bug to be patched. AppSec's threat model assumes the attacker controls some part of $I$ (SQL injection, XSS, SSRF) and tries to escape $f$'s intended domain. But an LLM does not have a well-defined domain. Its output space is open-ended by design, and the "intended domain" is a property negotiated in natural language between the deployer and the model—often without the security team in the room. Asking AppSec to "validate the output" is a category error: it is like asking a network firewall to validate the meaning of a sentence.

ML Security (ML安全) is built on a model-centric threat model: the adversary perturbs inputs (adversarial examples, 对抗样本), poisons training data (data poisoning, 数据投毒), or exfiltrates model parameters (model inversion/extraction, 模型反演/提取). These are real risks, but they assume the attacker interacts primarily with the model artifact. The 2023–2026 incident record shows that the majority of consequential failures occur at the *integration seam*, not inside the model. An LLM that produces perfectly safe completions in isolation can still be made to exfiltrate customer data when wrapped in a Retrieval-Augmented Generation (RAG, 检索增强生成) pipeline whose vector store is poisoned.

AI Alignment (AI对齐) addresses the most philosophically loaded question: does the model's objective match human intent? Methods like RLHF (Reinforcement Learning from Human Feedback, 人类反馈强化学习), Constitutional AI, and debate-based oversight try to shape the model's preferences during training. But Alignment does not produce deployable defenses. It does not give the engineer a way to *constrain* what the model can do at inference time; it tries to make the model *want* to do the right thing. As the 2024 literature increasingly recognizes, a well-aligned model wrapped in an unsafe tool-integration layer remains unsafe.

The blind spot. What falls between these three disciplines is the engineering practice of *building trustworthy integrations around non-deterministic components*. This practice has its own threat models (prompt injection, indirect prompt injection, agent hijacking, output-channel contamination), its own architectural patterns (capability-based isolation, output validators, dual-LLM patterns), its own testing methodology (red-teaming, jailbreak benchmarking, tool-call fuzzing), and its own runtime defenses (input filtering, output sanitization, action-policy engines). None of these belong cleanly to AppSec, MLSec, or Alignment. They belong to a fourth discipline: AI Security Engineering.

The seam problem is not new in spirit—any time a non-deterministic component meets a deterministic one, the seam has to be engineered (this is true of garbage-collected runtimes, of JIT compilers, of database query planners). What is new is that the seam is now *content-shaped* rather than *process-shaped*. The non-determinism is no longer a memory layout or a scheduling decision; it is the meaning of a sentence. And meaning cannot be inspected with a debugger. This shift forces a new discipline.

1.2 本体论：资产-威胁-能力-信任根四元组

A discipline without a clear object of study is a community, not a field. To make AI Security Engineering an engineering discipline rather than just a vocabulary, we need an ontology—a set of primitives that can describe every concrete security problem in the field. We propose the ATCR quartet (四元组): Assets, Threats, Capabilities, Roots of Trust.

This quartet is not arbitrary. It mirrors the structure of classical security engineering (cf. Anderson, *Security Engineering*; Shostack, *Threat Modeling*; the STRIDE and PASTA traditions), but is reshaped for AI-native systems where the trust boundary is content-shaped rather than process-shaped.

1.2.1 资产 (Assets)

In classical security, assets are data, services, and reputation. In AI Security Engineering, we must add three categories that classical security treats as second-class:

1. Model weights (模型权重) and their derived artifacts: adapters, LoRA matrices, embeddings, system prompts, tool schemas. These are first-class intellectual property with measurable market value and demonstrable exfiltration cost. 2. Prompt corpus and context windows (提示语料与上下文窗口): the effective "program" that an AI system runs is split between weights and prompts. A leaked system prompt or a poisoned few-shot example can be as damaging as a leaked source file. In agentic systems, the prompt is often the only audit artifact. 3. Action capability (行动能力): an LLM agent that can call `send_email(to=X, body=Y)`, `transfer_funds(account=Z, amount=N)`, or `db_query("DROP TABLE users")` possesses a kind of asset—*the ability to move the world*—that classical IT security struggles to inventory because it is created dynamically by the model at runtime. Action capability is an asset of the *system*, not the user.

The asset model must therefore be runtime-evaluated, not statically declared. A static list of "approved tools" is necessary but not sufficient; the model can compose tool calls in sequences the system integrator never imagined, and the resulting capability is a genuine, novel asset that did not exist at design time.

1.2.2 威胁 (Threats)

We adopt a four-axis threat taxonomy:

• **Axis 1 — Attack surface** (攻击面): prompt surface, tool surface, retrieval surface, training surface, deployment surface.
• **Axis 2 — Adversary location** (对手位置): user-channel (direct prompt injection, 直接提示注入), third-party-channel (indirect prompt injection via retrieved documents, emails, web pages — 间接提示注入), insider-channel (malicious fine-tuning data, poisoned RAG corpus), supply-chain (backdoored base models, compromised ML libraries).
• **Axis 3 — Adversary capability** (对手能力): naive user, skilled jailbreaker, well-resourced attacker with white-box model access, nation-state actor with pipeline access.
• **Axis 4 — Failure mode** (失效模式): information disclosure (信息泄露), integrity violation (完整性破坏), availabil