Chapter 7 — Hallucinations and Reliability

Seventh post of the chapter-by-chapter walkthrough of LLM Primer VII: AI Security. The chapter that treats reliability as a security property — because a confidently wrong output is a security problem whenever the consequences depend on correctness.

Why this chapter exists

Hallucinations were originally a natural-language-generation term for outputs that contained information not supported by their source. Applied to LLMs, the phenomenon splits into two useful senses: factuality failures, where the model contradicts established facts, and faithfulness failures, where the model drifts from user instructions or provided context. Both matter, and both have specific mechanisms. Ji et al.'s 2023 ACM Computing Surveys review and Huang et al.'s taxonomy in the same year gave the field its vocabulary. This chapter treats reliability as a first-class security property, because a system that acts on model output is only as safe as the output is correct — and the model has no built-in mechanism for knowing when it does not know.

7.1 Hallucinations have mechanisms, not moods

Language models predict tokens by sampling from a distribution the training objective placed mass on. The distribution is shaped by frequency in the training corpus, by the model's inductive biases, and by whatever alignment training was applied afterwards. None of these mechanisms selects for factual correctness in the way a database query does. Factuality failures happen when the model asserts something contradicted by the world — a wrong date, a fabricated citation, a non-existent function signature. Faithfulness failures happen when the model's output drifts from user intent or provided context — answering a slightly different question, ignoring parts of a passage the user asked to summarise, inconsistently reasoning across a long response. The two categories overlap but need different diagnostics. The mechanisms include next-token dynamics that trade fluency for accuracy in long generations, the training-data distribution's under-representation of niche facts, alignment training that discourages saying "I don't know," and prompt patterns that push the model toward plausibility rather than truth. Understanding the mechanism is what makes the mitigation targeted rather than performative.

7.2 Confidence is not correctness

A calibrated probabilistic system has stated confidences that match its accuracy: when it says 80%, it is right 80% of the time across a large sample. Guo and colleagues' 2017 ICML paper "On Calibration of Modern Neural Networks" reported a striking result — modern neural networks are systematically over-confident. A model saying 80% may be right 65%; a model saying 99% may be right 88%. The mechanism is the cross-entropy objective, which rewards placing mass on the correct class without penalising over-confidence. Larger and more expressive models fit training data more tightly, producing sharper distributions on test examples that superficially resemble training. For LLMs the problem is more pronounced because the outputs are token sequences, alignment training reshapes the distribution in ways that can move probabilities without moving accuracy, and users read a confident sentence as evidence the model knows what it is saying. The gap between confidence and correctness is the reason reliability engineering cannot just trust the model's own signal.

7.3 Calibration and hybrid verification are the operational fix

Calibration techniques divide into training-time and inference-time. Temperature scaling, introduced by Guo et al. in the same 2017 paper, is the standard: after training, adjust a single scalar that divides pre-softmax logits until calibration error is minimised on a held-out set. Predictions do not change; the probabilities do. Verbalised confidence — asking the model to output a confidence estimate alongside its answer — has been studied since 2022 and works to a degree, but the model's stated confidence is itself a language token subject to the same distributional pressures. Self-consistency sampling generates multiple completions and votes; agreement correlates with correctness better than any single completion's probability. Hybrid architectures do more work. Retrieval-augmented generation, as a reliability technique, has been the most consistently effective — the Vectara HHEM leaderboard has tracked well-configured RAG systems below 1% hallucination rate on factoid summarisation where bare generation exceeds 5% on the same tasks. Structured verification pipelines run the output through a second model that fact-checks against source documents. Human-in-the-loop review remains the strongest defence for high-stakes outputs, subject to the two failure modes the field has already named: rubber-stamp review at scale, and review-without-context where the reviewer lacks the source material to verify the model's claim.

What Chapter 7 sets up

Chapter 8 shifts from incidental failure modes to deliberate ones — adversarial attacks that treat the model as a target and construct inputs designed to manipulate outputs in directions the operator did not intend. The chapter walks the lineage from Goodfellow's 2014 FGSM through NLP-specific work — HotFlip, TextFooler, BERT-ATTACK — into the universal adversarial trigger work of Wallace and colleagues and the universal-suffix work of Zou and colleagues touched on in Chapter 4. It then walks black-box attacks against APIs and model-stealing, from Tramèr's 2016 USENIX paper through Carlini's 2024 ICML paper extracting production embedding-projection layers. Chapter 9 completes Part III with the supply chain: backdoored models, safetensors versus pickle, Sigstore signing, and drift monitoring.

Next — Chapter 8: Adversarial Attacks on Models. Gradient-based attacks in a discrete input space, black-box attacks through APIs, and model stealing as a confidentiality-plus-security concern.