Chapter 8 — Adversarial Attacks on Models

Eighth post of the chapter-by-chapter walkthrough of LLM Primer VII: AI Security. The chapter that traces adversarial attacks from Goodfellow's 2014 image-classifier work through TextFooler and universal suffixes to model stealing against production APIs.

Why this chapter exists

Chapter 4 treated prompt injection as the practical face of adversarial input in LLM systems. Chapter 8 walks the underlying research tradition. Goodfellow, Shlens, and Szegedy's 2014 paper "Explaining and Harnessing Adversarial Examples" argued that adversarial inputs are not a pathology but a consequence of models' near-linear behaviour in high-dimensional input spaces. That framing carried into NLP through work that solved the discrete-input problem — HotFlip, TextFooler, BERT-ATTACK — and into LLMs through universal adversarial triggers and, more recently, the Zou et al. universal-suffix work that transferred across closed-source APIs. Alongside adversarial input sits model stealing, a confidentiality attack whose extracted surrogate becomes an adversarial-input launchpad.

8.1 The lineage from FGSM to universal suffixes

Goodfellow's Fast Gradient Sign Method — perturb each input dimension by epsilon times the sign of the loss's gradient — was the canonical white-box attack for continuous inputs. Text resisted the approach because tokens are discrete: moving along the gradient of the embedding leaves the token space entirely. The NLP adversarial-example literature was largely about finding good discrete approximations. HotFlip (Ebrahimi et al., ACL 2018) used a single-character flip that most changed the loss. TextFooler (Jin et al., AAAI 2020) substituted synonyms with beam search under semantic-similarity constraints. BERT-ATTACK used a masked-language-model to propose candidate substitutions. Wallace et al.'s universal adversarial triggers found short token sequences that, prepended to arbitrary inputs, induced targeted misbehaviour. Zou et al.'s 2023 "Universal and Transferable Adversarial Attacks on Aligned Language Models" showed that gradient-optimised suffixes derived on open-source models transferred to closed-source ones queried through APIs — the theoretical distinction between white-box and black-box collapsed, in practice, because the attackers had white-box access to enough surrogate models to generate transferable attacks. That transferability is what made the universal-suffix work matter for production deployments that thought their API opacity was protective.

8.2 Black-box attacks are cheaper than the API budget suggests

The commercial LLMs of consequence do not expose weights. The relevant threat model is black-box: the attacker pays for API access, sends queries, observes responses, and refines. The literature has shown surprisingly strong attacks in this setting. Brute-force search over variants of a prompt handles small attack surfaces — short adversarial suffixes, single-word substitutions — and is the workhorse of practical jailbreaking. Query-efficient methods use the model's own output signal as a proxy for the gradient the attacker cannot compute directly: if the response shifts detectably when a token changes, the attacker can climb toward the target. Automated jailbreak generation — PAIR (Chao et al. 2023), TAP (Mehrotra et al. 2023) — uses an attacker LLM to propose refinements against feedback from the target. The economics matter. Query costs are pennies; the attacker's total spend to develop a working jailbreak is often under fifty dollars and produces an attack that generalises across users, sessions, and sometimes across model versions. That is a very different attacker profile from "someone with an academic GPU cluster."

8.3 Model stealing turns black-box into effective white-box

Model stealing — model extraction — is the class where the attacker's goal is not to manipulate a specific output but to reconstruct enough of the target's behaviour that they can use the reconstruction as a substitute. Tramèr and colleagues' 2016 USENIX Security paper "Stealing Machine Learning Models via Prediction APIs" established the research line against Amazon Machine Learning, BigML, and similar services. Krishna and colleagues' 2020 ICLR paper "Thieves on Sesame Street" showed extraction against BERT-style models. Carlini and colleagues' 2024 ICML paper "Stealing Part of a Production Language Model" demonstrated extraction of the embedding-projection layer of production models including OpenAI's through targeted API queries — a partial extraction that nonetheless revealed hidden dimensions and structural information the provider had not intended to release. The security consequence, beyond the confidentiality loss, is that an extracted surrogate is a white-box target for generating transferable adversarial examples against the original. The defensive layer is compositional: rate limiting on a per-account, per-key, per-IP, and per-tenant basis; anomaly detection on query patterns that suggest extraction (uniform distributions, systematic prompt variations, high-entropy outputs); and adversarial-input detection at the boundary. Watermarking research aims to make extracted models detectable, but the state of the art is still evolving.

What Chapter 8 sets up

Chapter 9 turns to a class of risk more fundamental than crafted inputs against a known-good model: attacks against the model itself before deployment, through the supply chain that produced it. An attacker who controls training data, model weights at any point between training and deployment, or dependencies at inference time has a stronger position than any input-space attacker. The chapter traces the BadNets line of Gu et al. from 2017 through its translation into LLMs, including Anthropic's 2024 "Sleeper Agents" research demonstrating trained-in backdoors persisting through safety training. It walks format-level risks — pickle deserialisation vulnerabilities catalogued as CVE-2024-3568 and adjacent entries, safetensors as the safer alternative — and the deployment-pipeline patterns (model signing, hash verification, SLSA, Sigstore) that production AI organisations have adopted to close the gap.

Next — Chapter 9: Model Integrity and Supply Chain Risks. BadNets, Sleeper Agents, pickle vs safetensors, Sigstore for model artefacts, and drift monitoring as the ongoing counterpart to deployment-time integrity.