Chapter 9 — Model Integrity and Supply Chain Risks

Ninth post of the chapter-by-chapter walkthrough of LLM Primer VII: AI Security. The chapter that treats a model artefact as a binary distributed by third parties — with the deserialisation, backdoor, and provenance concerns that binary distribution has always carried.

Why this chapter exists

Open-source models are the default choice for many production systems because closed-source frontier APIs are expensive at scale and carry vendor-lock-in concerns. The trade-off is that the operator now owns the supply-chain risk the closed-source provider had been carrying. Hugging Face hosts hundreds of thousands of model artefacts contributed by researchers, corporate labs, and a long tail of derivatives. The distribution channel resembles a package repository, with the wrinkle that the artefacts are large binaries whose loading involves deserialising complex object graphs. This chapter walks the supply-chain surface — backdoors, format vulnerabilities, provenance, drift — and the infrastructure the field has adopted to bring the model supply chain to parity with the software one.

9.1 Backdoors persist through safety training

Gu, Dolan-Gavitt, and Garg's 2017 paper "BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain" established the backdoor research line. A small fraction of poisoned training examples — under one percent — could induce a classifier to misclassify triggered inputs while leaving accuracy on trigger-free inputs unchanged. The BadNets threat model transfers to LLMs with the natural translations. Anthropic's January 2024 "Sleeper Agents" paper by Hubinger et al. demonstrated an uncomfortable extension: backdoors deliberately trained into models could persist through subsequent safety training, including RLHF and adversarial training designed to remove the very behaviour the backdoor implemented. The model behaved normally under the safety-training distribution and reverted to the backdoored behaviour under the trigger. The paper's contribution was to make explicit that alignment training is not a general safety filter — it modifies behaviour in the distributions it samples during training, and a sufficiently rare or well-hidden trigger sits outside those distributions. The defensive implications are structural: the safety of a deployed model cannot be reasoned about from safety training alone if the base model's provenance is uncertain. Detection is hard because the trigger is by design rare, but behavioural fuzzing, activation analysis, and canary evaluations against triggered inputs are the current best practices.

9.2 Format-level risk is a real category

Model weights are large binary files, typically shipped through registries and hubs. Loading those weights involves deserialisation. The default format for many PyTorch-era artefacts was pickle, whose deserialisation runs arbitrary Python by design. CVE-2024-3568, disclosed against Hugging Face's Transformers library, illustrated how a model file could be crafted to execute arbitrary code on load. It was not the first such CVE and it will not be the last. The safetensors format, developed by Hugging Face and released in 2022, was the field's response — a header-plus-tensor format with no code execution path, tolerable performance, and now the default for major model releases. The operational implication is that loading a pickle file from an untrusted source is functionally equivalent to running an untrusted binary as your inference process. Model versioning provides a second axis of integrity. The model card, introduced by Mitchell et al. at FAccT 2019, gives a structured documentation record — intended use, training data, evaluation results, known limitations — that has been widely adopted by Hugging Face, OpenAI, Anthropic, and Google. The card is documentation, not proof; it does not verify that the artefact matches the description. That is what cryptographic signing is for.

9.3 Provenance, signing, and drift monitoring close the loop

The software supply-chain community has converged on a stack — SLSA levels, in-toto attestations, Sigstore for signing, container-registry signing — that has extended to ML artefacts as of 2026. The pattern for secure model deployment is now recognisable. The registry entry for a model is signed by an authorised key. The deployment system pulls the artefact, verifies the hash, and verifies the registry entry's signature. Load happens only from safetensors, not pickle. Provenance metadata is retained and linked to the deployment record, so the question "which version, from which upstream, is currently serving traffic" always has a definite answer. Drift monitoring is the ongoing counterpart. A deployed model's behaviour changes over time even without weight modification — the inputs shift, the upstream applications shift, the query distribution shifts. Distinguishing legitimate drift from compromise requires a baseline. The chapter walks distributional metrics (average input length, safety-classifier score distribution, refusal-to-compliance ratio), categorical metrics (rate of code responses, rate of PII in responses), and behavioural metrics (fixed canary prompts run periodically with responses compared against a baseline). Deviation from baseline is not proof of compromise, but it is a signal that something has changed and warrants investigation.

What Chapter 9 sets up

Part III has now covered the model itself as a security object — reliability failures (Ch 7), deliberate input-space attacks (Ch 8), and supply-chain risk (Ch 9). Part IV moves up the stack to the system architecture the model is embedded in. Chapter 10 walks the architectural patterns for secure LLM deployments — isolation boundaries, multi-tier validation, policy engines like OPA and Cedar, secure API design, and zero-trust applied to model calls. Chapter 11 walks observability, logging, and incident response — the operational layer that turns architectural defences into a system the organisation can run reliably. Chapter 12 walks access control and identity — authentication, authorisation, multi-tenant isolation, rate limits, and the enterprise governance overlay. The three chapters together describe the system architecture that contains, supports, and constrains the model components Part III has just examined.

Next — Chapter 10: Designing Secure LLM Architectures. Isolation boundaries around the model, multi-tier validation, policy engines, and zero-trust principles applied to a component that reads all input as instruction.