Chapter 3 — Retrieval-Augmented Generation
Third post of the chapter-by-chapter walkthrough of LLM Primer V: Building Real-World LLM Applications. The chapter that walks the RAG pipeline end to end and separates the demo that works on your ten favorite documents from the system that survives contact with your real corpus.
Why this chapter exists
A foundation model knows what its training corpus showed it and nothing else. The product you are building almost certainly needs the model to reason about things outside that corpus — internal documents, last week's tickets, the customer's order history, a policy that shipped this morning. Retrieval-Augmented Generation is the engineering discipline that closes the gap: at query time, fetch the relevant material from a system of record, format it into the prompt, and let the model generate against it. The naive version is one embedding call and a top-k lookup. The production version is a pipeline with chunking strategy, query transformation, hybrid scoring, reranking, and an evaluation loop. Chapter 3 walks the pipeline end to end, and then the techniques that turn a demo pipeline into a deployment.
3.1 The five-stage pipeline and hybrid retrieval
The minimal RAG pipeline has five stages, and each interacts with the others in ways that are obvious in hindsight. Loaders preserve structure — headings, section paths, timestamps, source URLs, access-control labels — because everything downstream depends on what the loader kept. Chunkers cut along the document's natural grain, not along an arbitrary token count. Embedders project chunks into a vector space whose geometry is entirely determined by the embedding model's training distribution. Retrievers find nearest neighbors. Generators are prompted with the retrieved context, and the framing matters: "answer using only the context below, and respond 'I don't have that information' otherwise" is the single most effective hallucination-reduction pattern in production RAG. Pure dense retrieval understands paraphrase but misses exact identifiers; lexical retrieval catches identifiers but misses semantics; hybrid retrieval, fused with Reciprocal Rank Fusion, gets both effects, and a cross-encoder reranker over the fused top-50 buys another ten to twenty percent in precision.
3.2 Chunking is where quality lives or dies
The default of "split every 500 tokens with 50 tokens of overlap" works for a surprising fraction of generic corpora and fails for almost every specialized one. The structural chunker walks HTML or Markdown by heading level and emits one chunk per leaf section, prefixed with the breadcrumb of ancestor headings. The parent-child chunker embeds small child chunks for retrieval precision but expands each hit to its parent paragraph before passing it to the generator, decoupling retrieval unit from context unit. The semantic chunker walks a sentence-embedding sequence and splits where topic shifts. Composed, structural first and semantic within long sections, the two patterns handle almost every source type a real corpus contains. And every chunk carries enrichment metadata — source, URL, timestamp, heading path, language, visibility scope — because those are the fields that make retrieved chunks attributable, filterable, and legible to the rest of the system.
| Chunker | How it cuts | Best for | Watch-out |
|---|---|---|---|
| Fixed-size (default) | ~500 tok windows, ~50 tok overlap | Generic prose, prototypes | Breaks tables, code, lists |
| Structural | Along HTML/Markdown headings, one chunk per leaf | Docs, wikis, spec files | Needs breadcrumb prefix |
| Parent-child | Embed small child, return larger parent | Long-form articles | Two-tier store to maintain |
| Semantic | Split where sentence-embedding similarity drops | Narrative, transcripts | Slowest to build the index |
| Composed | Structural first, semantic within long leaves | Real production corpora | Recommended default |
3.3 Query transformation, multimodal, and text-to-SQL
The user's query is rarely the ideal query for retrieval. Multi-query expansion asks the model for several rephrasings, retrieves for each, and fuses. HyDE — Hypothetical Document Embeddings — asks the model to invent a plausible answer and embeds that instead of the question, on the theory that answers live in a different region of embedding space than questions do. Step-back prompting produces a more general version of the question, retrieves against both, and lets the model use the framing to answer the specific case. Decomposition splits a compound question into sub-questions the retriever can handle one at a time. Iterative retrieval lets the model decide what to fetch next — the bridge between RAG and agents. RAG also extends past text: CLIP-style joint image-text embedding spaces support multimodal retrieval, and text-to-SQL treats database schemas as the retrieval corpus and generates queries against a read replica with a timeout. A router picks the right transformation per query rather than running all of them.
| Transformation | What it does | When to route to it |
|---|---|---|
| Multi-query | Generate N rephrasings, retrieve each, fuse | Short, ambiguous queries |
| HyDE | Model invents a plausible answer; embed that | Question <> answer wording gap |
| Step-back | Generalize question, retrieve on both | Specific question needing framing |
| Decomposition | Split compound into sub-questions | "and", "vs", multi-hop questions |
| Iterative | Model decides what to fetch next | Bridge to agentic RAG |
| Multimodal (CLIP) | Joint image-text embedding | Screenshot / image queries |
| Text-to-SQL | Schema as corpus; generate SQL against read replica | Structured / analytical questions |
What Chapter 3 sets up
RAG is one capability among many. A production assistant rarely lives on retrieval alone: it needs to fetch a customer's recent orders, check inventory in another system, summarize the results, ask a clarifying question, and decide on its own when to do which. The natural framing for that behavior is agentic — the model selects from a set of tools, the system runs the chosen tool, the result returns to the model, and the loop continues until the task is done. Retrieval, in that framing, is one tool the agent can reach for. Chapter 4 turns the wrapper into an agent: the ReAct loop, tool schemas as contracts, and the three memory layers that let an agent hold state across turns.
