Chapter 6 — AI Observability and Tracing
Sixth post of the chapter-by-chapter walkthrough of LLM Primer V: Building Real-World LLM Applications. The chapter that treats a user query as a causal tree, not a request log, and shows what has to be traced for the tree to be legible.
Why this chapter exists
A single user query in an LLM system fans out into a tree: a query rewrite, three retrieval calls against two indexes, a rerank, a planner call, four tool invocations, a summarizer, and a final generation. Collapse that tree into "request received, response sent" and the truth is gone. When the user complains the answer was wrong, the team cannot say whether retrieval missed the right chunk, the planner chose the wrong tool, the tool returned the wrong data, or the generator ignored what it was given. Chapter 6 rebuilds observability for this shape. Classical distributed tracing is the substrate; the OpenTelemetry GenAI semantic conventions are the extension that makes model calls first-class citizens; LLM-specific metrics — TTFT, TPOT, cost per call, judge scores — are what turns a trace into something the team can operate against.
6.1 From request logs to causal traces
The move is from flat request logs to nested spans. OpenTelemetry with GenAI semantic conventions gives portable, provider-neutral span shapes for LLM calls, embeddings, retrievals, and tool invocations. The agent loop becomes a root span; each iteration becomes a child span; every LLM call, tool call, and retrieval nests underneath. Multi-turn sessions become trees of trees, with span links modeling state shared across turns — the summary from turn three that turn seven is still relying on. The value shows up the first time debugging: "which chunk did the retriever surface for that query, and did the generator actually cite it," a question that flat logs cannot answer without archaeology, becomes a single trace to look at. The span shape is portable across vendors because it is the OpenTelemetry standard, not a proprietary format, and switching from LangSmith to Langfuse becomes a routing change instead of a rewrite.
6.2 The metrics that matter — TTFT, TPOT, cost, quality
Total latency is not enough. Time-to-first-token dominates perceived speed for streamed responses — a user watches for the first character, not the last — and TTFT belongs on the span alongside total latency. Time-per-output-token dominates the feel of long responses and is what a user senses when they say "it was slow to finish." Cost belongs on the span as a first-class attribute, computed at the call site from a versioned pricing table so that the historical trace tells the truth even after prices change. Quality signals — a thumbs-down from the user, a judge score from the evaluation pipeline, a fallback trigger from the wrapper — attach to the trace as attributes, which is how the team correlates a bad answer with the specific model call and retrieval that produced it. The metrics are visible on dashboards, but the higher payoff is that they are queryable per trace, so a regression can be filtered by "high cost, low judge score" and the offending path surfaces immediately.
| Metric | What it captures | Why it beats total latency |
|---|---|---|
| TTFT (time-to-first-token) | Latency until first streamed byte | What perceived speed feels like |
| TPOT (time-per-output-token) | Steady-state token throughput | Why long responses feel slow |
| Cost per span | USD at call site, versioned pricing | Historical traces stay accurate after price changes |
| Quality attributes | Judge score, thumbs, fallback flag | Correlates a bad answer to a specific span |
| Tokens in/out | Prompt and completion counts | Drives cost, drives context-window ceilings |
6.3 Platforms and closing the loop
Four platforms occupy the current landscape. LangSmith is the LangChain-integrated choice with the least friction if the app is already in LangChain. Langfuse is the open-source self-hosted option that fits regulated deployments and organizations that need traces to stay inside their perimeter. Arize Phoenix has strong embedding analysis and is the right tool when the diagnostic question is "which queries are landing in the wrong region of the vector space." Helicone works as a proxy in front of the provider API, which is the lowest-touch integration but sees only the LLM call layer. The choice is operational, not qualitative — most of them expose the same core shape. The higher-value work is the export pipeline: negative-feedback traces, low judge scores, and fallback triggers become candidate evaluation cases, curated into the eval set, gated by the next release. That loop is what turns production into a teacher for development.
| Platform | Deployment | Strength | Fits when |
|---|---|---|---|
| LangSmith | SaaS | Native LangChain integration | App already lives in LangChain |
| Langfuse | Open-source, self-hosted | Perimeter control, OSS licensing | Regulated / air-gapped deployments |
| Arize Phoenix | SaaS or OSS | Embedding drift and cluster analysis | Diagnosing retrieval-space regressions |
| Helicone | API proxy | Zero-code integration | Only the LLM call layer needs visibility |
What Chapter 6 sets up
Traces are also where security incidents become visible. A successful prompt injection looks like an unexpected tool call in the trace tree. Exfiltration looks like a retrieval span whose chunk the model then obeyed as an instruction. A leaked system prompt appears as a completion span whose text overlaps improbably with the prompt above it. Chapter 7 turns the same tracing substrate toward the security discipline — the OWASP LLM Top 10 as a mid-2025 vocabulary, the direct-versus-indirect injection taxonomy that the vocabulary tries to name, and the four-layer mitigation matrix that enforces the principle that authority must match trust origin.
