Chapter 15 — Serverless APIs vs Dedicated Infrastructure

Fifteenth post of the chapter-by-chapter walkthrough of LLM Primer VI: Scaling AI Systems. The chapter that puts the break-even math on the table and then names the platform-engineering line item that pushes the answer for most teams.

Why this chapter exists

Chapter 14 left the team holding a clean picture of what the API bill is made of, and a quiet suspicion it could be cheaper. The natural next question is whether the team should be paying per token at all. The alternative is to bring inference in-house, run an open-weight model on the stack the earlier chapters described, and pay per GPU-hour instead. The trade has a clean break-even point. What is less clean, and what most teams underestimate, is everything on the dedicated side of the ledger that is not on the GPU invoice — the platform engineering, the security patches, the model-upgrade choreography, the 3 a.m. page when a node falls out of the fleet. Chapter 15 takes both sides seriously and derives when self-hosting is the right answer.

15.1 The break-even formula is clean, and mostly a decoy

The GPU produces throughput × utilization × 3,600 output tokens per hour. Its cost per million tokens is (gpu_hourly / tokens_per_hour) × 1,000,000. Break-even against the API is where that number equals the API's per-million price. A 70B model in FP8 on an H100 at $3/hour, running 800 tokens per second aggregate at 60 percent utilization, comes out to roughly $1 per million output tokens. The frontier API charges $15. The apparent savings are 14× — an argument for self-hosting that finance loves and that is missing the second, larger line item. The break-even also depends on utilization; below 20–30 percent utilization the self-hosted cost per million rises past the API rate and the GPU is more expensive than the alternative it was supposed to displace. Utilization, in turn, is a function of traffic shape, autoscaling quality, and whether the team can pack more than one workload onto the same fleet.

15.2 Platform engineering is the line item that decides the deal

The GPU invoice is the smaller half. The larger half is the team that keeps the fleet running: engines, drivers, orchestration, autoscaling, observability, security. The API team pays this cost implicitly through the per-token price; the self-hosting team pays it in headcount. A useful rule of thumb: a single serving GPU fleet — one or two model variants, autoscaling, observability, on-call, quarterly upgrades — requires roughly one full-time platform engineer plus a fraction of an SRE. A more diverse fleet — several engines, several models, multi-tenant scheduling — requires two to four. At $300k fully-loaded per engineer, the annual platform bill is $300k–$1.2M. The break-even monthly traffic where self-hosting pays back is somewhere between 200 million and 1 billion output tokens per month. Below that, engineering overhead exceeds token savings and the team should stay on an API. Above it, savings dominate. Platform engineering also comes in step functions, not smooth curves: the first hire brings the fleet up, the second happens because the first cannot cover weekend on-call, the third when the fleet grows to multi-region.

15.3 The realistic posture is hybrid, and the router is the load-bearing piece

The cleanest decision — all API or all dedicated — is rarely right for a team more than a year into production. Steady-state predictable workloads (overnight batch, scheduled reports, classification queues) pack well onto planned dedicated capacity. Bursty workloads (viral spikes, ticket floods, new-cohort launches) need API elasticity. Strict-latency workloads (interactive completion, voice) benefit from a colocated dedicated deployment. No-latency workloads (overnight enrichment, re-indexing) benefit from batch APIs at half price. The architecture that supports hybrid is a router at the application boundary that classifies each request by latency budget, complexity, and current backend utilization and dispatches to the cheapest backend that meets its SLA. A refined router also does within-API model routing (cheap model for simple queries, frontier for complex, reasoning tier for the ones that need it). The second-order benefit is optionality: workloads can migrate across the boundary by changing router policy, not application code. And the dedicated case often only pays back because a second workload — one that would otherwise have gone to the API — piggybacks on the already-paid-for GPU. A team that evaluates dedicated on a single workload usually never moves; a team that thinks in workload portfolios does, and saves.

What Chapter 15 sets up

Chapter 15 named a class of workload agents have introduced that fits neither "API call" nor "model serving": the execution environment for LLM-generated code, which lives on serverless functions, microVMs (Firecracker, gVisor), or WASM sandboxes depending on the isolation and package flexibility the agent needs. Chapter 16 closes the volume with the catalogue of cost-cutting moves that apply on either side of the posture line: intelligent model routing, application-level context compaction, batch APIs, semantic caching — the moves that compound into a third or a half of last month's bill.

Next — Chapter 16: Cost-Cutting Strategies in Production. The compounding moves that turn last month's bill into next month's savings.