Chapter 11 — Observability, Logging, and Incident Response

Published on: 2026-05-20 Last updated on: 2026-07-13 Version: 2
Chapter 11 — Observability, Logging, and Incident Response

Chapter 11 — Observability, Logging, and Incident Response

Eleventh post of the chapter-by-chapter walkthrough of LLM Primer VII: AI Security. The chapter that treats logging, alerting, and incident response as the layer that turns architectural defences into a system operators can actually run.

Why this chapter exists

A system with the right structure but no visibility fails invisibly. Chapter 11 walks the observability layer for LLM systems — what to log, how to detect abuse and anomaly, how to alert without producing noise, how to run an incident response playbook when something goes wrong, and how to learn from what happened. The general disciplines transfer from operational engineering elsewhere, with LLM-specific extensions: probabilistic outputs mean reproduction of incidents requires capturing more state than in deterministic systems; the OpenTelemetry GenAI semantic conventions, developed since 2024, give the vendor-neutral vocabulary for that state.

One line: In probabilistic systems, the absence of a complete log is the absence of a forensic record — and the log has to capture the exact model version, prompt, retrieved context, tool outputs, and sampling parameters that would let the interaction be reproduced.

11.1 What to log is a policy decision, not a default

Logging too little leaves the team unable to investigate. Logging too much creates compliance, cost, and privacy problems that eventually force reduction. The defensible position is the minimum that supports the operational use cases the team has identified, in a structured form that supports each. The purposes typically include real-time alerting on abuse or anomaly, after-the-fact investigation of incidents, capacity planning and cost analysis, compliance reporting, and continuous evaluation of model behaviour. Each purpose has different demands on the schema; a log designed for one is inadequate for the others. The OpenTelemetry GenAI semantic conventions define spans and attributes for the LLM call — model name, provider, request parameters, prompt content, response content, token counts, latency, cost — that let downstream tooling parse the same telemetry regardless of the specific SDK. Langfuse, Helicone, Arize Phoenix, and Datadog LLM Observability all consume this shape. A production log entry typically includes the request ID, the authenticated principal, the tenant, the model version and provider, the full assembled prompt (with retrieved context and its provenance), the tool calls and their outputs, the response content, the safety classifier scores at input and output, and the latency and token accounting. Storing this responsibly means being explicit about retention windows, PII handling, and access controls on the log store itself.

11.2 Detection composes signature, statistical, and behavioural signals

Once telemetry is structured, the next question is what patterns indicate something is wrong. Signature matching is the cheapest first line — known prompt-injection phrases, DAN-style preambles, base64-encoded payloads, role-play setups that have been observed before. The list is built from public research, internal red-team work, and past incidents. Signatures catch known variants; adversaries adapt once they learn which phrases are flagged. Statistical anomaly detection watches for divergence from baseline: unusual token distributions, atypical latency-to-length ratios, sudden spikes in refusal rates or in the rate of specific tool invocations. Baselines drift slowly under normal conditions and shift abruptly under abnormal ones. Behavioural pattern detection matches abuse profiles even when individual requests are not overtly malicious — a single principal issuing thousands of paraphrased variants of the same restricted request, a spike in requests that combine legitimate content with a specific suffix, a slow drift in per-user response distributions. Detection is only useful if it leads to alerts operators actually respond to. The taxonomy typically distinguishes critical (active abuse at scale, page an on-call engineer), high (notable pattern with bounded harm, business-hours notification), and medium/low (dashboards and weekly review). Alert fatigue is the failure mode; strict severity discipline is what prevents it.

11.3 Incident response is a playbook, not improvisation

NIST SP 800-61 Revision 2 gives the framework — preparation, detection and analysis, containment, eradication, recovery, post-incident activity — that the LLM-specific playbook extends. Preparation means the runbooks, on-call rotations, and access to the relevant tooling exist before the incident. Detection and analysis is where the observability of Section 11.1 pays off. Containment for an LLM incident may mean flipping a feature flag to disable a tool, downgrading to a more conservative model version, tightening rate limits on a specific principal or tenant, or rerouting traffic to an alternate stack. Eradication depends on the incident type: a jailbreak may need an added filter rule, a compromised RAG document may need to be removed from the index, a leaked credential may need rotation. Recovery is when the system returns to normal, with the containment reversed and the eradication verified. Post-incident activity is where the model-specific work concentrates: reproducing the behaviour if possible, characterising the boundary of the failure, deciding whether the incident reflects something about the model that should change how it is used, and feeding the outcome into the evaluation suite so future regressions are caught before deployment. Reproduction is not always possible under non-zero-temperature sampling, but the goal is to define when the unwanted behaviour occurs.

Worth holding onto: An incident investigation that cannot answer "which model version served this request, with what prompt, from what context" is an investigation that will end in speculation. Version pinning, prompt capture, and retrieval provenance in the log are what make root-cause analysis possible.

What Chapter 11 sets up

Chapter 12 closes Part IV with the identity and access layer — who is allowed to interact with the system, on what terms, and how the enforcement is structured across components. The traditional disciplines apply: authentication with API keys, OAuth, and mTLS; authorisation with RBAC and ABAC; multi-tenant isolation; rate limits and quotas; enterprise governance overlays. The LLM-specific extensions concern the model as a principal — an agent acting on behalf of a user has its own identity and its own permissions — the role of capability tokens for tool invocation, and the per-tenant configuration of model behaviour that multi-tenant LLM platforms have to support. Chapter 13 then opens Part V with the regulatory landscape, where the technical controls this book has developed have to be mapped to the AI Act, GDPR, the US state laws, and the frameworks that surround them.

Next — Chapter 12: Access Control and Identity. Authentication, RBAC vs ABAC, multi-tenant isolation, rate limits, and the enterprise governance overlay that makes LLM systems usable in regulated environments.

Want the full picture? The book chapter includes example OpenTelemetry GenAI span definitions, worked abuse-detection rules, sample incident-response playbooks, and the In Plain English sidebars this article only summarises. View LLM Primer VII on Amazon →
LLM Primer VII LLM observability Langfuse Arize Phoenix NIST SP 800-61 chapter 11
SHO
SHO
CTO of Receipt Roller Inc., he builds innovative AI solutions and writes to make large language models more understandable, sharing both practical uses and behind-the-scenes insights.