Chapter 2 — Threat Modeling for LLM Systems
Second post of the chapter-by-chapter walkthrough of LLM Primer VII: AI Security. The chapter that takes Shostack's four questions, STRIDE, PASTA, and MITRE ATLAS, and applies them to a system whose most powerful component reads all input as potentially instructional.
Why this chapter exists
Chapter 1 argued that AI security is structurally different. Chapter 2 gives that difference operational form. Adam Shostack's four questions — what are we working on, what can go wrong, what are we going to do about it, did we do a good job — are the same for any system, but the diagrams, asset inventories, and adversary catalogues that answer them look different when the system in question includes prompt-construction logic, a retrieval pipeline, a tool registry, and a probabilistic function that treats retrieved content on equal footing with the developer's instructions. The chapter walks the frameworks — STRIDE, PASTA, MITRE ATLAS, NIST AI 100-2 — and produces the working template of an LLM threat model that later chapters keep referring back to.
2.1 Frameworks adapt, diagrams cannot stay lazy
STRIDE — spoofing, tampering, repudiation, information disclosure, denial of service, elevation of privilege — turns out to map surprisingly well onto LLM systems. Spoofing becomes identity attacks on the API or user impersonation. Tampering becomes prompt injection, training-data poisoning, and retrieval-index manipulation. Repudiation becomes the disputes over who sent which prompt and who produced which output. Information disclosure becomes training-data extraction, system-prompt leakage, and cross-tenant exposure. Denial of service becomes OWASP's LLM10, unbounded consumption through expensive prompts and token floods. Elevation of privilege becomes the tool-use boundary — a user who induces the model to call a privileged tool inherits that tool's privileges. PASTA layers on business context and adversary simulation for teams that already do red-team work. Both frameworks assume a data-flow diagram that separates the components the surveillance question depends on. For LLM systems, the diagram should always separate the prompt-construction logic, the retrieval pipeline, the tool registry, the model call, the output-handling path, and the logging path.
| STRIDE letter | Manifestation in LLM systems | Where it usually lands |
|---|---|---|
| S — Spoofing | API key theft, user impersonation, forged tool callers | Ingress, identity plane |
| T — Tampering | Prompt injection, training-data poisoning, index manipulation | Prompt builder, RAG index |
| R — Repudiation | Disputes over who prompted what and who produced which output | Logging & audit trail |
| I — Information disclosure | Training-data extraction, system-prompt leakage, cross-tenant bleed | Model, prompt, output path |
| D — Denial of service | Unbounded consumption, expensive-prompt floods (OWASP LLM10) | Inference tier, rate limits |
| E — Elevation of privilege | User induces model to call privileged tool — inherits its scope | Tool registry, policy engine |
2.2 Assets that do not appear on conventional inventories
A threat model is only as good as its asset inventory. LLM systems introduce categories unfamiliar to teams whose previous work was on conventional applications. The model itself has several sub-assets — the weights (a multi-gigabyte binary that represents significant training investment), the documented behaviour (system prompt, safety policies, alignment training), and the reputation (a public failure damages the product independently of any technical compromise). The data covers training data, fine-tuning data, retrieval corpora, user inputs, and outputs; each has its own confidentiality, integrity, and availability requirements. The prompt itself is now an asset — the intellectual property of many products lives in a system prompt refined over months, and OWASP's 2025 list explicitly calls out system-prompt leakage as LLM07. Infrastructure covers the inference stack, the vector store, the tool interfaces, and the credentials that bind them. Logs are an asset because they are the forensic record, and second-order assets — model reputation, regulatory standing, customer trust — depend on the primary ones surviving contact with traffic.
2.3 Adversaries have specific incentives
A threat model that protects equally against everything protects against nothing. The adversary inventory has to be specific. Curious users probe the system to see what it will do — they use techniques from social media, their volume is high, their individual impact per incident is low, but their cumulative effect on the system's apparent safety is significant. Malicious users intend specific harm — extracting content the system should refuse, stealing other users' data or the system prompt, using the system to attack third parties through drafted phishing or generated malware. Competitors extract the model (Chapter 8) or the system prompt to reduce their own development cost. Insiders operate from inside the trust boundary. Nation-state actors combine model-level attacks with the wider tradecraft, and their targets are usually organisations rather than the model directly. Automated agents — themselves LLMs, sometimes driven by other adversaries — are the newest category and the one Chapter 17 takes up. Each adversary category has different capabilities, different incentives, and different detection profiles, and the mitigations that raise the cost against one may not affect another.
| MITRE ATLAS technique | Adversary type | What it targets & primary control |
|---|---|---|
| LLM Prompt Injection | Malicious user / attacker | Behaviour override — structured prompts, output filter |
| Jailbreak | Curious / malicious user | Safety envelope — layered guardrails, red-team CI |
| LLM Meta Prompt Extraction | Competitor | System prompt (LLM07) — assume leak, keep secrets out |
| Training Data Poisoning | Nation-state / supply chain | Weights & behaviour — data lineage, canary probes |
| Model Extraction | Competitor | Weights (via API) — rate limits, query anomaly detection |
| Membership Inference | Malicious user | Training data privacy — DP-SGD, dedup, temperature caps |
| ML Supply Chain Compromise | Insider / supply chain | Model artefact — Sigstore, safetensors, provenance |
| LLM Plugin Compromise | Malicious user / agent | Tool blast radius — least privilege, HITL confirmation |
What Chapter 2 sets up
The template developed here — one-page system description, data-flow diagram with trust boundaries, asset inventory, adversary catalogue, threat enumeration by STRIDE, mitigation mapping, residual-risk register — is the frame the rest of the book fills in. Chapter 3 expands the data asset category into its full structure — training-data risks, memorisation and extraction, sensitive-input handling, encryption and retention. Chapter 4 takes up prompt injection, which STRIDE's tampering category has already named as the dominant threat against the prompt-construction component. Chapters 5 and 6 develop the mitigations for prompt injection at the input, output, and RAG layers. Later chapters return to the same template — Chapter 11 for observability, Chapter 12 for identity — but the asset inventory and adversary catalogue introduced here are what those chapters extend.
