Chapter 6 — AI Observability and Tracing

Sixth post of the chapter-by-chapter walkthrough of LLM Primer V: Building Real-World LLM Applications. The chapter that treats a user query as a causal tree, not a request log, and shows what has to be traced for the tree to be legible.

Why this chapter exists

A single user query in an LLM system fans out into a tree: a query rewrite, three retrieval calls against two indexes, a rerank, a planner call, four tool invocations, a summarizer, and a final generation. Collapse that tree into "request received, response sent" and the truth is gone. When the user complains the answer was wrong, the team cannot say whether retrieval missed the right chunk, the planner chose the wrong tool, the tool returned the wrong data, or the generator ignored what it was given. Chapter 6 rebuilds observability for this shape. Classical distributed tracing is the substrate; the OpenTelemetry GenAI semantic conventions are the extension that makes model calls first-class citizens; LLM-specific metrics — TTFT, TPOT, cost per call, judge scores — are what turns a trace into something the team can operate against.

One line: LLM observability is distributed tracing plus GenAI semantic conventions plus a small set of metrics — TTFT, TPOT, cost, quality — and the export pipeline that turns production traces into the next release's evaluation set.

6.1 From request logs to causal traces

The move is from flat request logs to nested spans. OpenTelemetry with GenAI semantic conventions gives portable, provider-neutral span shapes for LLM calls, embeddings, retrievals, and tool invocations. The agent loop becomes a root span; each iteration becomes a child span; every LLM call, tool call, and retrieval nests underneath. Multi-turn sessions become trees of trees, with span links modeling state shared across turns — the summary from turn three that turn seven is still relying on. The value shows up the first time debugging: "which chunk did the retriever surface for that query, and did the generator actually cite it," a question that flat logs cannot answer without archaeology, becomes a single trace to look at. The span shape is portable across vendors because it is the OpenTelemetry standard, not a proprietary format, and switching from LangSmith to Langfuse becomes a routing change instead of a rewrite.

6.2 The metrics that matter — TTFT, TPOT, cost, quality

Total latency is not enough. Time-to-first-token dominates perceived speed for streamed responses — a user watches for the first character, not the last — and TTFT belongs on the span alongside total latency. Time-per-output-token dominates the feel of long responses and is what a user senses when they say "it was slow to finish." Cost belongs on the span as a first-class attribute, computed at the call site from a versioned pricing table so that the historical trace tells the truth even after prices change. Quality signals — a thumbs-down from the user, a judge score from the evaluation pipeline, a fallback trigger from the wrapper — attach to the trace as attributes, which is how the team correlates a bad answer with the specific model call and retrieval that produced it. The metrics are visible on dashboards, but the higher payoff is that they are queryable per trace, so a regression can be filtered by "high cost, low judge score" and the offending path surfaces immediately.

6.3 Platforms and closing the loop

Four platforms occupy the current landscape. LangSmith is the LangChain-integrated choice with the least friction if the app is already in LangChain. Langfuse is the open-source self-hosted option that fits regulated deployments and organizations that need traces to stay inside their perimeter. Arize Phoenix has strong embedding analysis and is the right tool when the diagnostic question is "which queries are landing in the wrong region of the vector space." Helicone works as a proxy in front of the provider API, which is the lowest-touch integration but sees only the LLM call layer. The choice is operational, not qualitative — most of them expose the same core shape. The higher-value work is the export pipeline: negative-feedback traces, low judge scores, and fallback triggers become candidate evaluation cases, curated into the eval set, gated by the next release. That loop is what turns production into a teacher for development.

Worth holding onto: The difference between "we know it broke" and "we know which step broke" is whether tracing was set up before the incident. Cost is just another metric on the trace, computed at the call site and stamped alongside latency.

What Chapter 6 sets up

Traces are also where security incidents become visible. A successful prompt injection looks like an unexpected tool call in the trace tree. Exfiltration looks like a retrieval span whose chunk the model then obeyed as an instruction. A leaked system prompt appears as a completion span whose text overlaps improbably with the prompt above it. Chapter 7 turns the same tracing substrate toward the security discipline — the OWASP LLM Top 10 as a mid-2025 vocabulary, the direct-versus-indirect injection taxonomy that the vocabulary tries to name, and the four-layer mitigation matrix that enforces the principle that authority must match trust origin.

Next — Chapter 7: LLM Security and Guardrails. The security posture that answers the question the tracing layer just made askable — what should have happened, and who was allowed to make that happen.

Want the full picture? The book chapter walks the OpenTelemetry GenAI span attributes in detail, the Langfuse and LangSmith SDK integrations with concrete code, the cost-attribution table with versioning, and the trace-to-eval export pipeline. View LLM Primer V on Amazon →