Chapter 13 — Autoscaling and Cold-Start Mitigation

Thirteenth post of the chapter-by-chapter walkthrough of LLM Primer VI: Scaling AI Systems. The chapter that explains why the default Kubernetes autoscaler produces outages under LLM traffic, and how KEDA, Knative, and CRIU compose the fix.

Why this chapter exists

A chat assistant swings 30× peak-to-trough across a day. A developer tool goes to zero on weekends. A consumer product rides a Pacific-to-European wave. Provisioning for peak wastes money in the trough; provisioning for trough drops requests in the peak. Autoscaling keeps the deployment in step with traffic; cold-start mitigation makes autoscaling — and particularly scale-to-zero — fast enough that users do not notice. Chapter 13 explains why the standard Kubernetes HPA is the wrong scaler for LLMs, what signals KEDA scales on instead, how Knative expresses scale-to-zero, and what it takes to compress a 60–180-second cold start into single-digit seconds.

One line: Scale on what the user feels — queue depth, time-to-first-token, KV occupancy — not on what the GPU driver reports, and pair the scaler with a CRIU-based warm-restore so the new replicas arrive before the spike is over.

13.1 HPA fails LLM traffic in four specific ways

Default HPA scales on CPU or GPU utilization. It fails LLMs in four ways. First, GPU utilization is the wrong signal — nvidia-smi reports the fraction of time some kernel is running, not whether the batch is well-packed or requests are making progress. A 95 percent-utilized server with eight long prefills queued behind one decode is overloaded; a 60 percent-utilized server with a healthy continuous batch is fine. Second, LLM request duration (seconds to tens of seconds) is long compared to HPA's control loop (15 seconds), so scaling lags the load by a full request lifetime. Third, the unit of capacity is not a pod; a disaggregated deployment scales in cliques and HPA does not understand cliques. Fourth, cold start: a new pod comes online 60–180 seconds later, long after the spike that triggered it has either overloaded the fleet or shed itself as timeouts. The typical failure mode is a two-minute outage during a traffic doubling, followed by a scale-down that leaves the deployment underprovisioned for the next spike.

13.2 KEDA scales on queue depth, TTFT, and KV occupancy

KEDA extends HPA with a ScaledObject CRD that binds a target workload to one or more scalers converting an external metric into a desired replica count. Three signals matter for LLM serving. Queue depth is the most direct: the engine's waiting-request count is the surplus of arriving work over serving capacity; vLLM exports it as vllm:num_requests_waiting. Time-to-first-token — specifically the p95 or p99 over a sliding window — captures user-visible degradation and catches asymmetric overload that queue depth misses (prefill saturated but decode fine, or vice versa). KV cache occupancy is forward-looking; when occupancy crosses 80 percent, new replicas should come online before the existing ones start preempting or rejecting. A production ScaledObject usually composes all three, with the most aggressive of them winning. Knative Serving layers on top for scale-to-zero: below a small activation threshold, all pods are removed; incoming requests are held by an activator that boots a pod on demand, which is only economical when the cold start is fast.

13.3 CRIU compresses a 90-second cold start into 3–6 seconds

A 70B cold start decomposes into image pull (10–60 s), Python init (2–5 s), weight load (60–120 s), CUDA context init (5–15 s), CUDA graph capture (10–30 s), KV warmup (2–10 s) — total 90–250 s. CRIU (Checkpoint/Restore In Userspace) snapshots a fully-warmed process — weights loaded, CUDA context up, graphs captured, KV pool allocated, health check passed — and restores it later from disk in seconds. NVIDIA's cuda-checkpoint utility (2024) extends CRIU to handle GPU state: device memory pools, CUDA context, cached PTX, captured graphs. A warm vLLM snapshot restores in 3–6 seconds from local NVMe. A DaemonSet pre-stages the checkpoint onto every node's NVMe so Knative can restore from local file rather than fetching over the network. Combined with image streaming (start pulling before the pull finishes) and lazy weight loading, sub-5-second cold starts are achievable in production — which is the number below which scale-to-zero becomes economically defensible for user-facing applications.

Worth holding onto: Scale-to-zero is the price of paying nothing at trough, and cold starts are the price of scale-to-zero. Every second you shave off cold start moves the break-even between "always warm" and "sometimes cold" in the operator's favor.

What Chapter 13 sets up

Chapters 1 through 13 walked the physical stack of LLM serving — hardware, engines, platforms, disaggregation, scaling. The rest of the book turns to the money. Chapter 14 explains why the token is the billing unit, why output is priced two to five times above input, and how invisible reasoning tokens end up on the invoice. Chapter 15 walks the break-even math between self-hosting and API providers, plus the platform-engineering line item most teams underestimate. Chapter 16 is the catalogue of cost-cutting moves that compound.

Next — Chapter 14: Token Economics and API Pricing. Why the invoice looks the way it does, and how the meter runs when no one is watching.

Want the full picture? The book chapter includes the KEDA ScaledObject YAML for a three-signal LLM scaler, the Knative Revision manifest for scale-to-zero, the full cold-start anatomy breakdown, the CRIU pre-stager DaemonSet, and the "day in the life" trace of an autoscaled deployment through a diurnal traffic curve. View LLM Primer VI on Amazon →