Chapter 16 — Cost-Cutting Strategies in Production

Final post of the chapter-by-chapter walkthrough of LLM Primer VI: Scaling AI Systems. The catalogue of orthogonal moves that compound into a third or a half of last month's bill.

Why this chapter exists

The previous two chapters told the team where the money goes and gave a framework for deciding which side of the API-versus-dedicated line to sit on. This chapter is the catalogue of moves the team can make once those decisions are settled. The moves are largely independent of posture — they save money on an API, they save money on dedicated hardware, they save money in a hybrid arrangement — and they compound. Teams that adopt them in sequence routinely find their inference bill, six months into a focused cost effort, is a third to a half of what it was. None of the moves are exotic; all of them require deliberate engineering; the cost of building them is small relative to what they save.

16.1 Intelligent model routing is the single largest lever

Production traffic is not uniformly difficult. Half or more of requests are simple enough for a budget-tier model an order of magnitude cheaper than the frontier; a smaller fraction genuinely need the frontier. The team that routes everything to the top overpays for the simple ones by 10–20×. The right architecture is a router that classifies each request and dispatches to the cheapest tier that can handle it. The cost gradient at time of writing is roughly 30:1 on input and 20:1 on output between frontier ($15–30/M output) and budget ($0.10–1/M). Router options range from a heuristic on request shape (length, format, tool-use markers) to a small budget-tier classifier that outputs a tier label to a learned router trained on historical which-model-actually-worked data. Design constraints: cheap and fast (under 100 ms, tenth of a cent), observable (every decision logged with features and outcome), safe by default (when uncertain, route up — the cost of over-route is one extra dollar, the cost of under-route is a visible quality regression).

16.2 Compaction, batch APIs, and semantic caching each buy back a large slice

Context compaction attacks the quadratic input growth Chapter 14 named. The simplest form is a rolling window: keep the last K turns verbatim, replace everything before with a budget-tier summary; the summary call costs a fraction of a cent and shrinks the input on every subsequent turn from thousands of verbatim history tokens to hundreds of summary. A more sophisticated compactor preserves specific kinds verbatim (code, tool outputs, user-provided data) and summarizes the rest. Asynchronous batch APIs at half price handle any workload without a synchronous latency budget — overnight summarization, re-classification, pre-generation for an A/B test, embedding refresh — the category is larger than teams initially recognize, and the engineering to move a workload to batch is trivial. Semantic caching answers before the model is called: embed each request, look up in a vector index of prior request-response pairs, return the cached response if similarity clears a threshold. Two users asking "what is the return policy" and "how do I return an item" hit the same entry. On FAQ-shaped traffic, cache hit rates of 30–50 percent are routine, and the cheapest token is the one never generated.

16.3 The moves compound multiplicatively, not additively

Take an agent whose bill has grown to $4,000/month at 1,000 queries/day. Route: 60 percent budget, 30 percent mid, 10 percent frontier → blended cost falls to 28 percent of frontier rate, bill drops to $1,100. Compact: input bloat cut in half, bill drops to $770. Move 20 percent to batch: $720. Semantic cache with 25 percent hit rate: $540. From $4,000 to $540 — an 86 percent reduction — with no user-visible change. Two more moves earn their keep even without their own sections. A prompt audit — reading every prompt with fresh eyes, deleting anything that cannot justify its tokens — typically finds 10–20 percent of spend to cut with no behavior change. An explicit max_tokens ceiling stops a response from running to 2,000 tokens when the application only needed 300 — nearly 7× overpayment on that request. The mental model is Book IV's: every request has a unit cost, and the engineering job is to route each request through cheap layers first and reach the expensive layer only when necessary.

What Chapter 16 closes

The volume opened with the autoregressive bottleneck — the irreducible fact that LLM generation is sequential and that the hardware which is so good at parallel arithmetic spends most of its time waiting for memory. Every chapter after that was a response to this central fact at a different level of the stack. Hardware chapters built the physical substrate. Model-level chapters shrank the model. Systems-level chapters extracted maximum useful work from each clock cycle. Engine and orchestration chapters put it into production. Economics chapters described how the team that runs the system pays for it. Every layer is a response to the previous one, and the discipline of running LLMs at scale is the integration of all of them by the same people, or at least by people who can talk to each other in the same vocabulary.

The book ends here. The series continues in Volume VII — AI Security, which extends the production-engineering discipline of this volume to threat modeling, guardrails, sandboxing of model-generated code, prompt-injection mitigation, output filtering, and the regulation that is now shaping how all of the above must be deployed. The infrastructure built across this volume — the inference fleet, the routing layer, the sandboxes, the agent execution environments — is exactly the infrastructure on which Volume VII's security concerns play out. Scaling and security are two sides of the same problem: one is the engineering of growth, the other is the engineering of resilience to the attacks that growth invites.