Chapter 2 — Foundation Models & Prompt Engineering
Second post of the chapter-by-chapter walkthrough of LLM Primer V: Building Real-World LLM Applications. The chapter that treats prompt engineering as engineering — versioned templates, defensive delimiters, and structured outputs — rather than as an art form best judged by vibes.
Why this chapter exists
Chapter 1 argued that the model is one component inside a deterministic wrapper. Chapter 2 looks hard at that component. The model catalog in 2026 is a taxonomy with sharp distinctions, not a list of equivalents. Sampling parameters turn the same weights into a deterministic classifier or a creative writer. Prompts are structured artifacts whose anatomy is the difference between surviving adversarial input and not. Mechanisms exist for forcing the output into a shape the wrapper can validate, and picking the wrong one for the task costs more in validation retries than the mechanism saved. A team that treats "the model" as a single anonymous resource, uses the defaults everywhere, writes prompts as throwaway strings, and validates output by hoping — that team has left most of the available engineering control on the floor.
2.1 Model selection as a first-class engineering decision
The catalog divides into four families: small language models tuned for speed and cost on bounded tasks; mid-tier general models that handle most production workloads; frontier models for the hardest reasoning and longest contexts; and reasoning models that spend additional inference compute on a hidden deliberation step. Multimodal cuts across all four. The selection is a per-request routing decision, not a global default: short queries to the SLM, non-trivial well-bounded tasks to mid-tier, hardest reasoning to the frontier, reasoning models where the latency is acceptable and the accuracy gain is real. The router logs its decision alongside the response, so a regression can be filtered by tier. Selection is also not a one-time decision — the model landscape moves faster than any other infrastructure the team depends on, and the discipline is to re-run the evaluation set quarterly and migrate when the evidence supports it.
2.2 Sampling as a deliberate profile
Temperature scales the token distribution before sampling; top-p truncates it to the nucleus above a probability threshold; min-p filters out tokens far below the top; seed offers best-effort repeatability. The two profiles worth naming explicitly are the deterministic profile — temperature zero, top-p 1.0, seed set — for classification, extraction, and routing tasks where the contract is "same input, same output"; and the creative profile — temperature around 0.8, top-p 0.95 — for generation where variation is the product. Mixing the two by accident, using the temperature-0.7 default on a classification task, produces the class of flaky tests and intermittent failures that teams blame on the model when the responsibility is the parameter choice. Streaming is orthogonal to sampling: the same parameters apply, the transport changes, and time-to-first-token replaces total latency as the perceived-speed metric.
2.3 Defensive prompts and structured outputs
A production prompt has a five-part anatomy — role, task, constraints, examples, delimited input — and the order matters because the model's attention to each component is influenced by what comes before it. Delimited input, anchored on both sides with the reminder that content between the markers is data and not instructions, is the floor below which a prompt should not fall. Prompts are versioned code artifacts; a name like prompts/classify_support_v3, logged in every trace, is how a regression gets traced back to the change that caused it. Structured outputs — Pydantic in Python, Zod in TypeScript, passed to the provider as a response schema — enforce the output's shape at decoding time. Provider-side JSON Schema enforcement removes an entire class of validation failures. Where JSON Schema is not expressive enough — SQL, regex-shaped formats, tool arguments against a closed catalog — grammar-constrained decoding via Outlines gives the same guarantee at the token level.
What Chapter 2 sets up
Chapter 2 assumed that whatever the prompt contains is enough for the task. Many tasks — classification, extraction, transformation — that assumption holds. For most user-facing production systems it fails, because the user is asking about facts the team owns and the model has never seen: internal documents, this week's policy, the customer's order history. The engineering move that closes that gap is retrieval-augmented generation. Chapter 3 walks the RAG pipeline end to end — loading, chunking, embedding, retrieving, generating — and then the techniques that separate a demo pipeline from a production one: hybrid retrieval, structure-aware chunking, and query transformations like HyDE and step-back.
Next — Chapter 3: Retrieval-Augmented Generation. The pipeline that gives the model the context its training data never covered — end to end, from loader to generator.
