Chapter 16 — Secure Fine-Tuning and Adaptation

Sixteenth post of the chapter-by-chapter walkthrough of LLM Primer VII: AI Security. The chapter that treats a fine-tuned model as an artefact whose security properties must be earned, not inherited — because the same gradient steps that teach domain vocabulary can also erode the alignment the base model arrived with.

Why this chapter exists

Fine-tuning looks like a low-risk operation. A team takes a well-aligned base model, tunes it on their domain data, and expects the aligned behaviour to survive. The empirical literature has been clear since 2023 that this expectation is wrong. Qi et al.'s 2024 ICLR paper "Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!" showed that even benign instruction data can lower refusal rates on harmfulness benchmarks. Yang et al.'s 2023 "Shadow Alignment" showed that a hundred deliberately crafted examples can subvert a safely-aligned open-weight model. The chapter walks the mechanisms, the poisoning threat model, the evaluation gates that catch regressions in CI, the alignment techniques that reinstall what tuning eroded, and the rollback discipline that turns bad updates into routine operations.

16.1 Alignment erodes through the mechanism that trains behaviour

Domain adaptation is usually well-motivated. The base model is too general, the team has a corpus that captures how their domain talks, tuning narrows behaviour into something more accurate and more on-brand. What gets missed is that narrowing is not free. The same gradient steps that teach domain vocabulary also re-weight every other behaviour, including the behaviours that kept the model from generating bioweapon synthesis or writing convincing phishing. Qi et al. measured this directly by fine-tuning a safety-tuned Llama-2 variant on the Alpaca instruction dataset — publicly available, no obviously harmful content — and observing that refusal rates on harmfulness benchmarks dropped substantially. Nothing in the training set asked the model to be less safe. The signal was carried by the absence of refusal demonstrations: the model learned that helpful answers were rewarded and refusing was rarely modelled, and it generalised that lesson onto requests where refusal had been the trained-in default. The mechanism is what gradient descent does when the objective is "follow the instruction" and the demonstrations almost never refuse. The mitigations are architectural — mix refusal examples into the fine-tuning set, use techniques like DPO or KTO that preserve reference-model behaviour, apply safety re-training after domain adaptation — but none of them is automatic.

16.2 Deliberate poisoning is a small-data attack

If accidental erosion is the common case, poisoning is the worst case. The threat model is straightforward: an attacker contributes some fraction of the fine-tuning data with the goal of installing a specific behaviour the deployer will not notice at evaluation. The behaviour can be a backdoor (triggered output on trigger phrase), a refusal lift (compliance where the base model refused), a content insertion (recommends a specific product or party when asked unrelated questions), or a long-horizon misalignment that activates under specific conditions. Yang et al.'s "Shadow Alignment" showed that a hundred adversarial pairs, structured as ordinary instruction-response, could subvert a major open-weight model's safety behaviour. Qi et al. at ICLR 2024 replicated at lower scale: roughly ten well-chosen examples were enough to materially compromise alignment. The examples did not need to be exotic. Mixed with benign data, the dataset looked unremarkable. The threat surface widens with the number of parties that contribute data — customer submissions to a labelling pipeline, contractors labelling data, employees preparing internal fine-tuning corpora, upstream open datasets. Each is a potential injection channel, and the defensive posture is provenance discipline: every training example has to be traceable to its origin, and the origin has to be trusted at the level the fine-tuning outcome requires.

16.3 Evaluation gates and rollback are the operational safety net

A fine-tuned checkpoint is not a deployable model. It is a candidate. The gap is closed by evaluation — both capability evaluation that confirms the model still does its job and safety evaluation that confirms it has not regressed. The right mental model is a deployment gate, not a benchmark report. A gate has pass/fail criteria, thresholds set in advance, and a defined consequence when criteria are not met — no promotion to staging, no traffic, automatic ticket. Under deadline pressure, benchmark reports become advisory; gates enforced by CI are what actually hold. The alignment techniques that restore safety after tuning erosion include mixing refusal examples into the fine-tuning set, RLHF or DPO on refusal preferences, Constitutional AI (Bai et al., Anthropic, 2022) as a training-time approach that scales without human labelling, and safety-tuned continued training on canary sets. Every fine-tuned model will misbehave eventually. The only question is whether the team can roll back to a known-good version in minutes or spends a day in incident response. The difference is discipline before deployment. A model registry — MLflow Model Registry, AWS SageMaker, Vertex AI, or in-house equivalents — that tracks every artefact, its provenance, its evaluation results, its deployment status, and its lineage is the foundational pattern. A registry that does not record evaluation results is an artefact store; an artefact store does not help during a rollback. The rollback itself needs to be a single command whose safety has been rehearsed.

What Chapter 16 sets up

Chapter 17 closes the book by looking forward to threats that are still emerging in mid-2026: autonomous agents that compose model output with tool use and run for hundreds of steps without supervision; multimodal models whose input surface now includes images and audio; synthetic identities that erode the assumption that the entity on the other end of a channel is who they claim; and the AI-versus-AI arms race in which both attackers and defenders are themselves models. The fine-tuning concerns from this chapter do not go away in that world; they intensify. A model whose alignment quietly drifted three weeks ago becomes a much bigger problem when it is also the orchestrator of an agent with shell access.

Next — Chapter 17: Future Threats and Emerging Defenses. Autonomous agents, multimodal attack surfaces, synthetic identity, AI-versus-AI dynamics, and the shape of AI assurance as the discipline it is becoming.