Chapter 3 — Retrieval-Augmented Generation

Third post of the chapter-by-chapter walkthrough of LLM Primer V: Building Real-World LLM Applications. The chapter that walks the RAG pipeline end to end and separates the demo that works on your ten favorite documents from the system that survives contact with your real corpus.

Why this chapter exists

A foundation model knows what its training corpus showed it and nothing else. The product you are building almost certainly needs the model to reason about things outside that corpus — internal documents, last week's tickets, the customer's order history, a policy that shipped this morning. Retrieval-Augmented Generation is the engineering discipline that closes the gap: at query time, fetch the relevant material from a system of record, format it into the prompt, and let the model generate against it. The naive version is one embedding call and a top-k lookup. The production version is a pipeline with chunking strategy, query transformation, hybrid scoring, reranking, and an evaluation loop. Chapter 3 walks the pipeline end to end, and then the techniques that turn a demo pipeline into a deployment.

One line: RAG is a five-stage pipeline — load, chunk, embed, retrieve, generate — where almost every quality complaint traced to its root turns out to be a chunking complaint in disguise, and hybrid retrieval plus a reranker is the shape production converges on.

3.1 The five-stage pipeline and hybrid retrieval

The minimal RAG pipeline has five stages, and each interacts with the others in ways that are obvious in hindsight. Loaders preserve structure — headings, section paths, timestamps, source URLs, access-control labels — because everything downstream depends on what the loader kept. Chunkers cut along the document's natural grain, not along an arbitrary token count. Embedders project chunks into a vector space whose geometry is entirely determined by the embedding model's training distribution. Retrievers find nearest neighbors. Generators are prompted with the retrieved context, and the framing matters: "answer using only the context below, and respond 'I don't have that information' otherwise" is the single most effective hallucination-reduction pattern in production RAG. Pure dense retrieval understands paraphrase but misses exact identifiers; lexical retrieval catches identifiers but misses semantics; hybrid retrieval, fused with Reciprocal Rank Fusion, gets both effects, and a cross-encoder reranker over the fused top-50 buys another ten to twenty percent in precision.

3.2 Chunking is where quality lives or dies

The default of "split every 500 tokens with 50 tokens of overlap" works for a surprising fraction of generic corpora and fails for almost every specialized one. The structural chunker walks HTML or Markdown by heading level and emits one chunk per leaf section, prefixed with the breadcrumb of ancestor headings. The parent-child chunker embeds small child chunks for retrieval precision but expands each hit to its parent paragraph before passing it to the generator, decoupling retrieval unit from context unit. The semantic chunker walks a sentence-embedding sequence and splits where topic shifts. Composed, structural first and semantic within long sections, the two patterns handle almost every source type a real corpus contains. And every chunk carries enrichment metadata — source, URL, timestamp, heading path, language, visibility scope — because those are the fields that make retrieved chunks attributable, filterable, and legible to the rest of the system.

3.3 Query transformation, multimodal, and text-to-SQL

The user's query is rarely the ideal query for retrieval. Multi-query expansion asks the model for several rephrasings, retrieves for each, and fuses. HyDE — Hypothetical Document Embeddings — asks the model to invent a plausible answer and embeds that instead of the question, on the theory that answers live in a different region of embedding space than questions do. Step-back prompting produces a more general version of the question, retrieves against both, and lets the model use the framing to answer the specific case. Decomposition splits a compound question into sub-questions the retriever can handle one at a time. Iterative retrieval lets the model decide what to fetch next — the bridge between RAG and agents. RAG also extends past text: CLIP-style joint image-text embedding spaces support multimodal retrieval, and text-to-SQL treats database schemas as the retrieval corpus and generates queries against a read replica with a timeout. A router picks the right transformation per query rather than running all of them.

Worth holding onto: A wrong retrieval rarely fails loudly — the model dutifully writes a confident answer from the wrong context, and quality erodes without anyone noticing. Instrument the retrieval stage before you tune it.

What Chapter 3 sets up

RAG is one capability among many. A production assistant rarely lives on retrieval alone: it needs to fetch a customer's recent orders, check inventory in another system, summarize the results, ask a clarifying question, and decide on its own when to do which. The natural framing for that behavior is agentic — the model selects from a set of tools, the system runs the chosen tool, the result returns to the model, and the loop continues until the task is done. Retrieval, in that framing, is one tool the agent can reach for. Chapter 4 turns the wrapper into an agent: the ReAct loop, tool schemas as contracts, and the three memory layers that let an agent hold state across turns.

Next — Chapter 4: AI Agents and Tool Calling. The loop that turns a stateless model into an actor pursuing a goal — with tools, memory, and hard bounds on runaway behavior.

Want the full picture? The book chapter includes the full hybrid retriever, the HTML-aware structural chunker, the parent-child store, the CLIP dual-embedder pattern, and the text-to-SQL pipeline with query validation. Vol III went deeper on RAG mechanics; Vol V summarizes them and moves on. View LLM Primer V on Amazon →