Chapter 11 — Continuous Updates and Pipeline Optimization

Eleventh and final post of the chapter-by-chapter walkthrough of LLM Primer III: Enhancing Enterprise AI with RAG. In which the pipeline is never finished — the documents change, the queries shift, the models are swapped — and the team that owns it learns to think in three timescales at once.

Why this chapter exists

A RAG system is not finished when it ships. The documents change, the queries drift, the model itself is replaced every few months. The pipeline a team is proud of in March is, by September, retrieving against stale embeddings produced by a two-generation-old model, serving a base model that has been quietly swapped, and answering a question distribution that has drifted in ways nobody charted. This chapter is about the engineering of staying current — detecting what changed in the corpus, keeping the index fresh without rebuilding it, keeping latency from creeping up, and closing the loop between production telemetry and the changes the team actually makes.

11.1 Change Data Capture and incremental indexing

The first instinct of every team that ships a RAG system is to schedule a nightly rebuild of the index. It works. It is also wrong long-term. A nightly rebuild burns embedding API calls on documents that did not change, leaves the index up to twenty-four hours stale, and stops fitting inside the nightly window as the corpus grows. The mature pattern is incremental indexing driven by Change Data Capture — the pipeline reacts to events from upstream rather than polling.

Three kinds of events matter. Insert: a new document, parsed, chunked, embedded, indexed. Update: an existing document changed; re-embed the affected chunks. Delete: a document removed; evict the corresponding vectors before they can return in results — a hard requirement under GDPR, CCPA, and the rest. The mechanism that makes this tractable is the content hash. On first ingest, store SHA-256 over the normalized chunk text alongside the embedding. On update, re-chunk, hash, and compare: unchanged chunks stay, new ones are embedded, old ones go. A paragraph edit becomes one embedding call, not a thousand. The embedding bill scales with editorial activity instead of with the corpus.

The harder problem is ordering and consistency. Events arrive out of order; a delete can race ahead of the update it should have followed. The standard remedy is monotonic versions per document, with conditional writes: only apply an event if its version exceeds the version on file. This makes the pipeline idempotent — a duplicate event cannot corrupt the index — which is not an optimization but a correctness requirement at scale. Compliance contexts add tombstones: a logical delete that takes effect at query time before the physical removal completes asynchronously, so the deletion is honored immediately.

11.2 Managing latency: semantic caching and model tiering

A retrieval-augmented call accumulates latency at every hop. The defense is to do less work when the work is provably unnecessary, and two techniques carry most of that weight. A conventional cache stores answers keyed by exact query text and hits a vanishing fraction of real traffic. Semantic caching keys by meaning instead: embed each incoming query, search a small cache of recent queries, return the cached answer if the cosine similarity to the nearest entry exceeds a threshold. "What is our refund policy?" and "how do refunds work?" share no string match and share the entire answer.

The three choices that matter are the similarity threshold (0.93–0.97 cosine for general embeddings, tuned against held-out traffic), the time-to-live (ideally tied to the contributing chunks — invalidate when any of them is re-embedded), and scope (partitioned by tenant, by access role, by anything that could leak one user's answer to another). Production deployments report 30–60% hit rates with tens of milliseconds on hits versus multi-second uncached responses, with proportional cost savings since cache hits skip both embedding and generation.

Model tiering handles the queries that must use the model and should not use the largest one available. Two or three tiers: a small fast cheap model for the bulk, a larger one for queries the small model is not confident on, optionally a third for the long tail. The router is where production deployments get this wrong on the first pass. The simplest version uses query-side signals (short factoid vs analytical). A better version uses retrieval-side signals (high consistent similarity means the small model is enough). The most sophisticated runs the small model first and escalates on calibrated confidence — accurate at the margin, paying for an extra inference on the escalations. The numbers to watch are escalation rate and regret rate together; either one alone misleads.

11.3 The continuous feedback loop

A RAG pipeline emits constant telemetry. Most teams collect it; very few close the loop on it. The loop has four stages. Collect: every query, retrieval, generation, citation, and user interaction logged with a stable schema and a stable query identifier that threads through every stage — without that identifier, diagnosing a regression devolves to guesswork. Evaluate: the Triad from Chapters 9 and 10 run in two modes, sampled offline against a reference set for accuracy, and online behavioral proxies (regenerate, copy, follow-up, abandonment) for coverage. Neither is sufficient alone. Together they triangulate.

Decide is the hardest stage, because the same signal implies several different remedies. A drop in Context Relevance might mean the corpus is missing topics, or the reranker has degraded, or the embedding model is no longer suited to the new vocabulary. Distinguishing requires slicing the metrics — by topic, by document age, by embedding version, by tenant — and the team that only watches the aggregate will discover, a quarter late, that one slice has been dragging the average down all along.

Apply comes in three weights. Configuration changes — top-k, reranker weights, hybrid alpha, cache threshold, escalation rule — A/B-tested in hours, rolled back in minutes, several running at any given time. Reindexing actions — re-embed a stale topic, ingest a new source, evict obsolete documents — weekly to on-demand, on a non-production replica before promotion. Model changes — swap the embedding model, swap the base model, retrain a custom reranker — quarterly, with shadow deployment, parallel evaluation, gradual traffic shift, and rollback option. The discipline is in the cadence, not any single change. A small human label channel — perhaps a hundred examples a week from the ambiguous-proxy queue — keeps the LLM judges calibrated and stops the loop from optimizing against its own proxies.

What Volume III sets up

We started by treating retrieval-augmented generation as the engineering answer to two problems pure language models cannot solve: fresh knowledge and verifiable provenance. We traced the architecture from the early embed-retrieve-stuff pattern through the modular and agentic systems now in production, and we did the careful work on every component along the way — the parsers that recover structure from PDFs, the chunkers that decide what a unit of meaning is, the vector databases that store the result, the hybrid retrievers and rerankers that find what matters, the access controls and anonymization layers that keep the system honest, the evaluation frameworks that tell us whether any of it works, and now the continuous-update mechanisms that keep it alive.

RAG is not a product category. It is an engineering discipline composed of perhaps a dozen sub-disciplines, each of which can be the difference between a system that works and one that quietly hallucinates. A team that treats each sub-discipline seriously will produce something that holds up under real traffic. A team that treats any of them as a black box will produce something that does not.

The series continues in LLM Primer IV — Designing AI Cognition with MCP. Where Volume III was about bringing the right knowledge to the model, Volume IV is about bringing the right hands — the Model Context Protocol, the agents that wield it, the tools they call, and the memory they accumulate. Same architectural sensibility, different surface of the same problem. The work continues.