Chapter 1 — Why AI Security Is Different

First post of the chapter-by-chapter walkthrough of LLM Primer VII: AI Security. The chapter that argues AI security is not just traditional security with an ML adjective attached — the substrate has changed, and every later chapter follows from that change.

Why this chapter exists

For three decades, security engineering has rested on a stable foundation: code and data are different, vulnerabilities are discrepancies between specified and actual behaviour, and patches close them. Large language models break this foundation in a specific way. The behaviour that needs defending is encoded not in source code but in billions of learned weights, executed against inputs that mix trusted instructions with untrusted content in the same string. The "vulnerability" is often not a bug — it is the model doing exactly what it was trained to do, in a context the designers did not anticipate. There is no patch for "the model was too helpful." There is only redesign, retraining, or additional containment. This chapter names the structural differences that shape everything that follows.

1.1 The substrate has changed

Traditional application security works because behaviour is specified in code and defects are locatable. SQL injection has a structural fix — parameterised queries — because there is a syntactic distinction between query and parameter. A language model has no such specification. It has a training objective and a distribution of weights, and its behaviour on any specific input is emergent. When a model refuses one phrasing and complies with another, there is no line to fix. The security question shifts from "is there a bug in this code path" to "what is this system capable of, and under what conditions does that capability become dangerous?" Traditional security also assumed determinism as the baseline; here the baseline is probabilistic. A safety filter that succeeds on a thousand test cases can fail on the thousand-and-first because sampling drew a different path. The defender reasons in distributions and confidence intervals, not in proofs of unreachability. The OWASP Top 10 for LLM Applications, revised in 2025, is one attempt to name this new layer — with prompt injection at LLM01 and unbounded consumption at LLM10 — but it is a floor above the existing web-application floor, not a replacement for it.

1.2 The attack surface widens

An LLM-integrated application introduces surfaces that did not exist before. The prompt itself is a concatenation of developer instructions, retrieved context, user input, and tool outputs — all as tokens the model reads without native trust boundaries. If the user can influence any portion, they share the same channel as the developer. The retrieval path is the second new surface: every document indexed becomes an indirect input, and anyone who can influence what enters the index can influence what the model sees. Greshake and colleagues named this indirect prompt injection in 2023 and showed the channel was both real and hard to close. The tool-use boundary is the third: each tool granted to the model is a privilege whose consequences leave the response text and reach real systems. The training pipeline is the fourth, since any data used to update the model becomes part of the trust boundary. The model artefact is the fifth — large binaries whose deserialisation, as CVE-2024-3568 demonstrated, can execute code on load. Output handling is the sixth, since model-generated content forwarded downstream is untrusted input by another name. MITRE ATLAS catalogues the tactics and techniques against this widened surface.

1.3 Models are becoming infrastructure

Between 2012 and 2022, models were features inside applications. A recommendation system that failed produced worse recommendations. Large language models, especially with tool use, have changed this. The model is increasingly the orchestration layer — reading documents, deciding which tool to call, drafting the message, generating the code another component executes. It is often the most powerful component in the system, and it is also the most malleable, driven by natural-language inputs anyone can author. A traditional database has query languages and access controls; an LLM acting as an orchestrator has none of these intrinsic constraints, only the ones the surrounding application has added. This is what "infrastructure" means here: load-bearing components whose compromise propagates. Infrastructure gets defined SLOs, comprehensive logging, change control, and incident response. Most LLM deployments through 2024 and 2025 had not yet reached that maturity. The infrastructure framing also reaches procurement: when an organisation embeds a managed LLM service into its stack, the vendor's model-update discipline, evaluation gates, and disclosure practices become part of the buyer's risk profile.

What Chapter 1 sets up

The rest of the book is a response to the structural shifts named here. Chapter 2 introduces threat modelling adapted to LLM systems — the STRIDE and PASTA frameworks turned toward assets, adversaries, and attack surfaces that do not appear on conventional diagrams. Chapter 3 works through the data dimension across its full lifecycle. Chapters 4–6 walk the prompt-and-interaction interior: injection, filtering, and RAG. Chapters 7–9 walk the model layer. Chapters 10–12 walk the system architecture around it. Chapters 13–15 add the regulatory, responsible-AI, and organisational perimeter. Chapter 16 walks fine-tuning as its own security surface, and Chapter 17 looks at the threats still forming. The whole arc rests on the premise this chapter has established: the substrate has changed, and the discipline has to change with it.

Next — Chapter 2: Threat Modeling for LLM Systems. Shostack's four questions, STRIDE and PASTA against LLM assets, and MITRE ATLAS as the tactics catalogue for adversaries this new surface attracts.