Chapter 12 — Disaggregated Serving and Kubernetes
Twelfth post of the chapter-by-chapter walkthrough of LLM Primer VI: Scaling AI Systems. The chapter that finally splits prefill and decode onto separate GPU pools and shows the Kubernetes primitives that keep the pods on the right side of the interconnect.
Why this chapter exists
Chapter 11 drew a clean line between engines and platforms — until the engine itself points out that prefill and decode are contradictory workloads. Prefill is compute-bound; decode is memory-bandwidth-bound. Running them on the same GPU forces one piece of hardware to be good at two contradictory things and it ends up mediocre at both. Disaggregated serving splits them onto separate GPU pools, ships the KV cache between the pools, and lets each pool be tuned for its own workload. The price is operational complexity, paid in Kubernetes primitives, pod topology, and the network fabric that carries the KV traffic. Chapter 12 walks the shape of that complexity — the components, the transfer path, and the CRDs (LeaderWorkerSet, Grove's PodCliqueSet, KAI Scheduler) that express it.
12.1 Prefill and decode want different chips at different clocks
On the same H100, a long prefill can genuinely reach 60–80 percent of peak FLOPs, and a decode step uses 5–15 percent of FLOPs while saturating 70–90 percent of HBM bandwidth. Interleaving them under continuous batching is a workable compromise but leaks in two ways. Prefill's compute pass hits the same HBM bus that queued decodes are trying to stream over, and time-between-tokens inflates. The card also runs at a single operating point: tune for prefill and decode is bandwidth-throttled; tune for decode and prefill is compute-throttled. Prompts have lengthened from hundreds to thousands of tokens as RAG and agents have taken over production traffic, and the interleaving cost has grown with them. Disaggregation gives each phase its own pool. Prefill workers can run on raw-FLOPs cards (H100 SXM, B200). Decode workers can run on capacity-and-bandwidth cards (H200, MI300X). Each pool is sized to its own workload — long-context, short-output traffic needs more prefill; chat traffic needs more decode.
12.2 The shape is four components plus a KV transfer path
A working disaggregated deployment has four components. Prefill workers accept a request, compute prefill, emit the first token, and stage the KV cache for transfer. Decode workers accept an incoming KV cache and its metadata, install it into their paged-attention pool, and run continuous batching over their in-flight generations. A KV cache router — the control plane — maintains a view of every decode worker's free KV capacity, batch fullness, and network locality, and assigns each prefill's output to minimize transfer cost and balance load. A frontend gateway terminates the user's HTTP connection, streams the first token from the prefill worker, then transparently switches to streaming the rest from the decode worker. The critical constraint is the KV transfer: a 70B GQA sequence at 4,096 tokens is 1.5–2 GB, and it has to move from prefill node to decode node inside the same 50–100 ms TTFT budget the user is watching. NVLink at 900 GB/s inside a node and InfiniBand at 400 Gb/s between nodes make the numbers work — if the pods land on the right side of the fabric.
12.3 LeaderWorkerSet, Grove, and KAI express the topology
Kubernetes did not originally have a primitive for "these two typed pods are one logical replica." LeaderWorkerSet, added upstream in 2024, expresses a multi-pod replica with a leader (say prefill) and workers (say decodes); the controller keeps the group as one scheduling unit. NVIDIA Grove's PodCliqueSet (2025, part of NVIDIA AI Enterprise) generalizes further with typed cliques (prefill, decode, router), each with its own template and size, plus a clique-topology description of how the cliques relate. Grove submits the whole set to the scheduler together with intra-clique and inter-clique locality constraints. Neither CRD is enough on its own; both need a scheduler that knows the cluster's physical topology, which the default Kubernetes scheduler does not. KAI Scheduler — open-sourced from Run:ai in 2024 — consumes a topology graph produced by nvidia-smi topo --matrix and the InfiniBand subnet manager, and scores candidate placements against it. When a Grove clique requests nvlinkDomain: required, KAI restricts placement to a single NVLink-coherent domain (an HGX baseboard's eight GPUs on one NVSwitch). When inter-clique topology says sameInfiniBandIsland, KAI keeps the cliques within one leaf-switch group where KV round-trip stays under 100 μs.
What Chapter 12 sets up
Chapter 12 gave the deployment a static picture: at any moment, a fixed number of cliques are running and serving traffic. Real traffic is not static — chat assistants swing 30× peak-to-trough across a day, developer tools go to zero on weekends, consumer products chase the sun. Chapter 13 walks the autoscaling story: why standard HPA is the wrong scaler for LLM serving, what signals KEDA scales on instead, how Knative expresses scale-to-zero, and what compresses a 60–180-second cold start into something a user-facing application can absorb.
Next — Chapter 13: Autoscaling and Cold-Start Mitigation. KEDA, Knative, CRIU, CUDA graph caching, NVMe — the stack that lets scale-to-zero coexist with real users.
