Chapter 13 — Autoscaling and Cold-Start Mitigation
Thirteenth post of the chapter-by-chapter walkthrough of LLM Primer VI: Scaling AI Systems. The chapter that explains why the default Kubernetes autoscaler produces outages under LLM traffic, and how KEDA, Knative, and CRIU compose the fix.
Why this chapter exists
A chat assistant swings 30× peak-to-trough across a day. A developer tool goes to zero on weekends. A consumer product rides a Pacific-to-European wave. Provisioning for peak wastes money in the trough; provisioning for trough drops requests in the peak. Autoscaling keeps the deployment in step with traffic; cold-start mitigation makes autoscaling — and particularly scale-to-zero — fast enough that users do not notice. Chapter 13 explains why the standard Kubernetes HPA is the wrong scaler for LLMs, what signals KEDA scales on instead, how Knative expresses scale-to-zero, and what it takes to compress a 60–180-second cold start into single-digit seconds.
13.1 HPA fails LLM traffic in four specific ways
Default HPA scales on CPU or GPU utilization. It fails LLMs in four ways. First, GPU utilization is the wrong signal — nvidia-smi reports the fraction of time some kernel is running, not whether the batch is well-packed or requests are making progress. A 95 percent-utilized server with eight long prefills queued behind one decode is overloaded; a 60 percent-utilized server with a healthy continuous batch is fine. Second, LLM request duration (seconds to tens of seconds) is long compared to HPA's control loop (15 seconds), so scaling lags the load by a full request lifetime. Third, the unit of capacity is not a pod; a disaggregated deployment scales in cliques and HPA does not understand cliques. Fourth, cold start: a new pod comes online 60–180 seconds later, long after the spike that triggered it has either overloaded the fleet or shed itself as timeouts. The typical failure mode is a two-minute outage during a traffic doubling, followed by a scale-down that leaves the deployment underprovisioned for the next spike.
| Failure mode | HPA behavior | Why LLMs break it |
|---|---|---|
| Wrong signal | Scales on GPU / CPU utilization | 95% util can mean queued prefills, not healthy load |
| Slow control loop | 15 s reconcile | LLM requests take 5–30 s — scaler lags by a full request |
| Wrong unit | Scales pod count | Disaggregated deployments scale cliques, not pods |
| Cold start | Waits for new pod to become ready | 60–180 s to warm 70B; spike is over by then |
13.2 KEDA scales on queue depth, TTFT, and KV occupancy
KEDA extends HPA with a ScaledObject CRD that binds a target workload to one or more scalers converting an external metric into a desired replica count. Three signals matter for LLM serving. Queue depth is the most direct: the engine's waiting-request count is the surplus of arriving work over serving capacity; vLLM exports it as vllm:num_requests_waiting. Time-to-first-token — specifically the p95 or p99 over a sliding window — captures user-visible degradation and catches asymmetric overload that queue depth misses (prefill saturated but decode fine, or vice versa). KV cache occupancy is forward-looking; when occupancy crosses 80 percent, new replicas should come online before the existing ones start preempting or rejecting. A production ScaledObject usually composes all three, with the most aggressive of them winning. Knative Serving layers on top for scale-to-zero: below a small activation threshold, all pods are removed; incoming requests are held by an activator that boots a pod on demand, which is only economical when the cold start is fast.
| Signal | Metric | Threshold | Catches |
|---|---|---|---|
| Queue depth | vllm:num_requests_waiting | > batch-size / N | Arriving work > serving capacity |
| TTFT p95 | sliding-window latency | > 200 ms | User-visible degradation, asymmetric overload |
| KV occupancy | paged-attention block usage | > 80% | Impending preemption / rejection |
13.3 CRIU compresses a 90-second cold start into 3–6 seconds
A 70B cold start decomposes into image pull (10–60 s), Python init (2–5 s), weight load (60–120 s), CUDA context init (5–15 s), CUDA graph capture (10–30 s), KV warmup (2–10 s) — total 90–250 s. CRIU (Checkpoint/Restore In Userspace) snapshots a fully-warmed process — weights loaded, CUDA context up, graphs captured, KV pool allocated, health check passed — and restores it later from disk in seconds. NVIDIA's cuda-checkpoint utility (2024) extends CRIU to handle GPU state: device memory pools, CUDA context, cached PTX, captured graphs. A warm vLLM snapshot restores in 3–6 seconds from local NVMe. A DaemonSet pre-stages the checkpoint onto every node's NVMe so Knative can restore from local file rather than fetching over the network. Combined with image streaming (start pulling before the pull finishes) and lazy weight loading, sub-5-second cold starts are achievable in production — which is the number below which scale-to-zero becomes economically defensible for user-facing applications.
What Chapter 13 sets up
Chapters 1 through 13 walked the physical stack of LLM serving — hardware, engines, platforms, disaggregation, scaling. The rest of the book turns to the money. Chapter 14 explains why the token is the billing unit, why output is priced two to five times above input, and how invisible reasoning tokens end up on the invoice. Chapter 15 walks the break-even math between self-hosting and API providers, plus the platform-engineering line item most teams underestimate. Chapter 16 is the catalogue of cost-cutting moves that compound.
