Chapter 5 — Input Validation and Output Filtering
Fifth post of the chapter-by-chapter walkthrough of LLM Primer VII: AI Security. The chapter that turns the layered mitigation frame of Chapter 4 into operational discipline — sanitisation stages, guardrail tooling, structured output, red teaming, and safety metrics that mean something.
Why this chapter exists
The four-layer mitigation architecture named in Chapter 4 is only as good as its layers are operational. Chapter 5 develops two of them: the input side, where user requests are inspected and either passed, transformed, or refused; and the output side, where the model's response is checked against a second set of gates before it leaves the system. Around both sit the disciplines of structured prompting, adversarial testing, and safety measurement. The tools have matured — Llama Guard, NeMo Guardrails, Lakera Guard, AWS Bedrock Guardrails, Garak, PyRIT, promptfoo — and the operational patterns for wiring them together have converged.
5.1 Sanitisation is staged, not single-shot
"Sanitising input" carries misleading connotations from the SQL-injection era. In LLM systems, sanitisation is a staged process — inspect, classify, transform, pass or refuse — whose output is not a "safe" input but an input that has passed a set of policy gates with the outcome recorded. The pipeline typically has four kinds of check, ordered by cost. Structural checks are cheapest: length limits, character-set constraints, Unicode NFKC normalisation, stripping of zero-width and confusable characters that make up much of the smuggling literature. Pattern-based checks catch explicit overrides and known adversarial templates; they are noisy in both directions but useful as a coarse screen when hits are logged rather than blocked. Classification checks use dedicated safety models — Llama Guard, the OpenAI Moderation API, and the equivalents from Lakera and AWS Bedrock — to score the input against a defined taxonomy. LLM-based checks, the most expensive tier, invoke a smaller model to reason about intent when the classifier's confidence is intermediate. Each stage has a false-positive and a false-negative rate, and both need to be measured before the pipeline is trusted with production traffic.
5.2 Structured output is a defence, not just a formatting convenience
The defence-in-depth architecture of Chapter 4 has a third structural layer that this chapter develops: constraining the model's output to a defined schema, so that even an output influenced by injection cannot escape the structural envelope. The simplest form is schema-enforced JSON. The model is instructed to emit JSON conforming to a schema; the application parses and validates; non-conforming output is rejected or retried. Jason Liu's instructor library wraps the OpenAI and Anthropic clients with pydantic models — the developer writes a class describing the output structure and the library handles the prompt construction, validation, and retry. Microsoft Research's Guidance goes further, constraining token-by-token generation against a template that defines exactly which positions can contain free text. On the output-moderation side, Meta's Llama Guard family — versions 1 through 3, with multimodal coverage added in 3 — has become the canonical open-weight classifier over the 2023–2025 window, integrated into most production stacks as the response filter. NVIDIA NeMo Guardrails and the commercial offerings from Lakera, AWS Bedrock, and Cisco AI Defense compete on similar footing.
5.3 A defence not measured is not a defence
Red teaming is what turns a safety claim into a measurement. Manual red teaming — trained adversarial testers, often external — produces specific prompts that succeeded against the deployment, grouped by attack pattern. Automated red teaming scales the manual work across the input space. NVIDIA Garak, open-sourced in 2023 and continuously updated, runs a battery of probes against a target endpoint and reports which succeeded; probes cover prompt injection, data leakage, hate-speech elicitation, encoding smuggling, role-play jailbreaks, and so on. Microsoft PyRIT (Python Risk Identification Toolkit), released 2024, adds an agentic red-team pattern in which one model generates attacks against another. promptfoo compares prompts and models against evaluation sets, useful when the question is which configuration is safer. The metrics that matter compose two failure modes. Attack success rate answers "what fraction of a defined attack set gets through?" Refusal calibration answers "what fraction of refused requests should not have been refused?" A system with zero attack success and 50% refusal rate has not solved the problem; it has shifted the cost from unsafe outputs to unhelpful ones. Both metrics require labelled samples drawn from real traffic distributions, and both are conditional on evaluation set composition. Reporting a single number without the composition is where safety claims most often mislead.
What Chapter 5 sets up
Chapter 6 takes up retrieval-augmented generation specifically. The input layer developed here treats the user's message as the untrusted portion. RAG systems add a second untrusted portion: the retrieved chunks, whose provenance is often less clean than the user's message. Greshake's indirect prompt injection, Liu's characterisation of injection attacks against LLM-integrated applications, Zhong's poisoning of retrieval corpora, and the newer PoisonedRAG and BadRAG lines all describe how this second surface fails. Chapter 6 walks the trust boundaries in RAG, the specific attack patterns, the secure-retrieval architecture the field has converged on, and the monitoring practices that surface retrieval-level attacks before they become incidents.
Next — Chapter 6: Retrieval-Augmented Generation Risks. Trust boundaries in RAG, malicious document injection, index poisoning through the embedding path, and the monitoring that catches attacks the sanitisation missed.
