Chapter 15 — Building a Secure AI Organization

Fifteenth post of the chapter-by-chapter walkthrough of LLM Primer VII: AI Security. The chapter that treats security culture, red teams, vendor risk, and long-term stewardship as the organisational infrastructure that carries the discipline over years.

Why this chapter exists

Technical controls without organisational discipline do not survive contact with time. Chapter 15 walks the layer where security culture, red-team practice, vendor risk assessment, continuous evaluation, and long-term stewardship live. The premise is that AI systems are part of the security perimeter rather than tools used within it — the model itself can be attacked, manipulated, or extracted, and its behaviour can be a vector for downstream attacks. The organisational infrastructure has to reflect this. The chapter draws on the published responsible-scaling frameworks — Anthropic, OpenAI, DeepMind, Microsoft, Meta — as the industry floor and works out what maintaining that floor requires from teams and structures.

15.1 Culture, red teams, and internal audit set the operating floor

Security culture is the shared set of attitudes through which an organisation's members address security in their daily work. It is hard to engineer directly; it is the downstream property of structures, incentives, and stories. For AI teams the culture has to recognise that the model itself is part of the perimeter and that AI-specific failure modes — prompt injection, hallucination, alignment erosion — are the team's responsibility rather than someone else's. Red teams give the culture its measurement. Microsoft's AI Red Team, established 2018, has been a notable public contributor, and the PyRIT framework released 2024 gave the field concrete tooling. Internal red teams differ from traditional ones — the inputs are natural language rather than crafted exploits, the attack surface is behaviour rather than code, the success criterion is model output rather than system compromise — but the discipline is the same. Coverage across prompt injection, jailbreaks, harmful content elicitation, bias probes, privacy leakage, and factual errors is the current expected scope. External red teaming complements internal for high-impact applications. Internal audit closes the loop by verifying that the controls the organisation says it has are the controls that are actually in place — the same discipline that has served information security for decades, applied to a new class of asset.

15.2 Vendor risk assessment is the supply-chain layer

Modern AI systems are built from components: foundation models from one provider, fine-tuning infrastructure from another, evaluation tools from a third, vector databases from a fourth, observability platforms from a fifth. The supply chain is long, the components are heterogeneous, and the failure of any one of them can compromise the whole. Vendor risk assessment is the discipline of evaluating the risks the supply chain introduces and managing them. The starting point is inventory — an organisation that does not know which AI vendors it depends on cannot assess the risks those vendors introduce. The inventory captures services consumed, data flows involved, contractual terms, certifications held (SOC 2 Type II, ISO/IEC 27001, ISO/IEC 42001 where available), public information about security posture, and criticality to operations. From the inventory, the assessment work follows: reviewing SOC 2 and ISO reports, examining data-handling commitments, evaluating incident-response track records, testing the vendor's own security claims, and monitoring for signals that the vendor's posture has changed. The ISO/IEC 42001 AI management system standard, published 2023, is becoming the natural focal point for vendor certification in AI, complementing the general information-security certifications the field already uses.

15.3 Continuous evaluation and long-term stewardship close the loop

Pre-deployment evaluation is a snapshot. Continuous evaluation is the operating discipline that keeps the snapshot from going stale. Stanford HELM provides public infrastructure for continuous capability and fairness evaluation across models, and the resulting dashboards let organisations benchmark their deployed models against external references. For internal use, the continuous evaluation infrastructure includes canary prompts run periodically with baseline comparison, red-team probes run on schedule and after model updates, safety benchmarks re-run to catch regressions, and production sampling for human review. The Anthropic Responsible Scaling Policy, the OpenAI Preparedness Framework, and the DeepMind Frontier Safety Framework each specify triggers and thresholds that require additional evaluation when specific capability milestones are approached. Long-term stewardship extends the discipline over years. Models have a lifecycle — development, evaluation, deployment, operation, update, deprecation. Each transition has stewardship requirements: development produces documentation and initial evaluation; deployment produces operating commitments; operation produces logs and evaluation; update produces new versions with their own documentation; deprecation produces end-of-life handling. The cross-cutting discipline that maintains continuity across phases is what "stewardship" names, and it is the layer that separates organisations that operate AI responsibly at horizon of years from those that operate it responsibly at horizon of quarters.

What Chapter 15 sets up

Chapter 16 narrows to fine-tuning as its own security surface. The chapter treats the fine-tuned model as an artefact whose security properties must be earned, not inherited. Even benign fine-tuning data can erode the base model's alignment, as Qi et al. demonstrated in the 2024 ICLR paper "Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!" Deliberate poisoning — Yang et al.'s 2023 "Shadow Alignment" — turns the same mechanism into an attack. The chapter walks the alignment erosion mechanism, the poisoning threat model, the CI evaluation gates that catch regressions before deployment, the alignment techniques (RLHF, DPO, Constitutional AI, RLAIF) that reinstall what tuning eroded, and the rollback discipline that turns a bad update into a five-minute incident rather than a day of firefighting. Chapter 17 then closes the volume with the emerging threats still forming.

Next — Chapter 16: Secure Fine-Tuning and Adaptation. Alignment erosion through benign data, deliberate poisoning, evaluation gates that stop bad checkpoints, and the model registry that makes rollback a routine operation.