Chapter 1 — The Mechanics of Token Generation

First post of the chapter-by-chapter walkthrough of LLM Primer VI: Scaling AI Systems. The chapter that argues almost every hard question in LLM serving descends from a single fact — the loop that produces each token is memory-bandwidth-bound, and the expensive compute you paid for sits idle 99.7 percent of the time.

Why this chapter exists

An LLM is, in operation, a next-token predictor wrapped in a loop. Tokens go in, a probability distribution over the next token comes out, one is picked, appended, and the loop runs again. Every interesting property of an inference stack — batching, quantization, KV caching, speculative decoding, disaggregated serving — descends from a close look inside that loop. The loop hides two workloads that share a code path but stress the hardware in opposite ways. One is compute-bound. The other is bandwidth-bound. On the same H100. In the same forward pass. Naming the split precisely is the first move of the book, and it is the frame every later chapter refers back to.

1.1 The autoregressive loop is sequential by force

Token t+1 is a function of every token up to and including t. The model cannot predict t+2 until t+1 has been sampled, because the prediction of t+2 requires t+1 to be part of the input. There is no clever kernel that parallelizes the generation of two consecutive tokens for a single sequence; the sequential character is enforced by the dependency structure of the computation. The wall-clock cost of an N-token completion is therefore N times the cost of one step plus fixed overhead. Every optimization that follows in the book — larger batches, speculation ahead of the trajectory, cheaper per-step arithmetic — is a partial answer to the question "given that the loop must walk one token at a time, how do we make each step faster or each batch larger?" The model also has no external scratchpad: it thinks by emitting tokens. The loop is the only mechanism by which the model gets to think for longer, which is why chain-of-thought and speculative decoding both live inside the same accounting.

1.2 Prefill and decoding stress the chip in opposite ways

The loop hides two phases. Prefill is the first forward pass, which consumes the user's prompt at shape [batch, sequence_length, hidden_dim]. Every matrix multiplication operates over all sequence positions at once; arithmetic scales with sequence length; the weights are read from HBM once and applied to many rows of work. Arithmetic intensity is high. Prefill is compute-bound and gets to use the H100's 989 BF16 TFLOP/s. Decoding is every subsequent forward pass. The input shape collapses to [batch, 1, hidden_dim]. Each layer's weights still have to be streamed from HBM, but only one row of arithmetic is performed against them. Arithmetic intensity crashes by three orders of magnitude. Decoding is memory-bandwidth-bound. On a 70B model in BF16, prefill on a 2,000-token prompt runs near the chip's compute roofline; the very next forward pass — the first decode step — runs against the same 140 GB of weights to produce one token. The chip did not change. The workload did.

1.3 A single user leaves 99.7 percent of the H100 idle

The consequence is that single-user real-time generation is the worst case for a frontier accelerator. A 70B model in BF16 on one H100 SXM decodes at roughly 24 tokens per second — brisk reading speed for the user, and the chip's HBM3 bandwidth of 3.35 TB/s is fully saturated moving 140 GB of weights per token. But at that rate the compute units perform only about 3.36 TFLOP/s of the 989 they are capable of. Compute utilization is 0.34 percent. The physical chip is not misconfigured; every SM is reading and multiplying at full speed, but the tensor cores are designed to consume tiles of many rows against each weight tile, and a single-token forward pass gives them one row. The compute capacity is denominated in a unit (operations per loaded byte) the workload does not produce. Someone still pays four to eight dollars an hour for the whole chip. The economics of LLM serving are therefore the economics of finding work for the parked compute — via batching, via cheaper per-token bandwidth, via speculation, via hardware whose balance point matches the workload.

What Chapter 1 sets up

The rest of the book is the response to the asymmetry named here. Chapter 2 takes apart the KV cache — the data structure that lets decoding avoid quadratic recomputation and, in the process, becomes the largest single consumer of VRAM in a serving cluster. Chapters 3 and 4 walk the hardware substrate with the bandwidth-versus-compute lens. Chapters 5 and 6 shrink the per-token bandwidth burden by shrinking the weights. Chapter 7 walks batching as the system-level lever that converts bandwidth-bound waste into throughput. Chapters 8 and 9 change the shape of the work with paged KV management and speculative decoding. Each move is legible as an attempt to fill in the idle compute this chapter has just measured.

Next — Chapter 2: The KV Cache Challenge. The workspace that makes decoding affordable, and the memory formula that explains why serving clusters run out of VRAM before they run out of compute.