Chapter 3 — Data Center GPUs for Generative AI

Third post of the chapter-by-chapter walkthrough of LLM Primer VI: Scaling AI Systems. The chapter that argues you should buy a serving GPU by its HBM bandwidth and VRAM capacity, not by the FLOP/s number on the front of the datasheet.

Why this chapter exists

Chapters 1 and 2 established the workload: prefill wants matrix engines saturated, decoding wants weights and KV cache streamed as fast as possible, and the KV cache footprint scales multiplicatively. Silicon selection is where those two profiles meet the market. The data-center GPU catalogue in 2026 is a small set of cards that look superficially similar — an accelerator, a few thousand cores, a bank of on-package memory — and diverge sharply in the two dimensions that actually govern decoding cost: HBM bandwidth and VRAM capacity. Chapter 3 walks the lineup that dominates production today — H100, H200, B200, L40S, MI300X — and gives an engineer enough mechanism to pick a card for a given workload rather than trust a benchmark chart.

One line: For serving, the two spec-sheet numbers that matter are HBM bandwidth (how fast weights and KV stream) and VRAM (how big a model fits before you must go multi-GPU) — the FLOP/s number mostly matters for prefill.

3.1 The H100 is the safe default, the H200 is the bandwidth patch

The H100 remains the workhorse of nearly every production inference fleet without a specific reason to be on something else. 80 GB of HBM3, 3.35 TB/s of bandwidth on the SXM variant (2.04 TB/s on PCIe), 989 BF16 TFLOPs, and — crucially — native FP8 in the tensor cores, which halves the per-token weight traffic and lets a 70B model fit in 70 GB rather than 140. The software stack is mature; the pricing has settled. Where the H100 strains is on capacity: 80 GB caps a 70B FP8 model plus a serving-batch KV cache and forces tensor parallelism above that. The H200 is the surgical fix — same Hopper compute, same 700 W envelope, but 141 GB of HBM3e and 4.80 TB/s of bandwidth. For any workload that was HBM-limited on H100 (which is almost every decoding workload), H200 is a drop-in throughput uplift of roughly 40 percent from bandwidth alone, plus headroom for the KV cache the H100 could not hold.

3.2 Blackwell adds FP4 and doubles the bandwidth ceiling

The B200 is not a Hopper refresh. 192 GB of HBM3e, 8.00 TB/s of bandwidth, 2250 BF16 TFLOPs, and a second-generation transformer engine that supports FP4 in the tensor cores natively. FP4 halves per-token weight traffic once more relative to FP8; combined with the raw bandwidth doubling, a 70B model that decodes at ~24 tokens per second on H100 comfortably exceeds 100 tokens per second on B200 for a single user, and scales roughly linearly with batch. The 192 GB capacity holds a 180B model in FP8 on one card and a 70B in FP16 with substantial KV headroom, which removes tensor parallelism from a lot of deployments. The cost is the price and the 1000 W thermal envelope; the workloads where Blackwell pays back most obviously are the ones where H100 was fighting a bandwidth or capacity ceiling on every request.

3.3 L40S and MI300X are the shape-of-workload choices

The L40S is the Ada-generation card for workloads that do not need a data-center part. 48 GB of GDDR6, 0.86 TB/s of bandwidth, and roughly a quarter of the H100's HBM throughput — but a third of the price and a standard PCIe form factor. For small models (up to 13B or so), for asynchronous batch workloads, or for edge deployments where a data-center HGX chassis is not available, L40S is the right answer and buying an H100 for it is waste. The AMD MI300X is the other end of the spectrum: 192 GB of HBM3, 5.30 TB/s of bandwidth, and 1307 BF16 TFLOPs on OAM. In raw serving terms it is competitive with H200 or better on capacity, roughly matches on bandwidth-limited decoding, and ships at meaningfully lower unit cost. The trade is the software stack — ROCm has closed most of the gap versus CUDA but not all of it, and the mature engines (vLLM, TensorRT-LLM, SGLang) are more battle-tested on NVIDIA. For teams with the ROCm competence, MI300X is the cost lever; for teams without it, the H200 is the safer buy.

Worth holding onto: Read the datasheet in this order: VRAM (does the model fit?), HBM bandwidth (how fast does decoding run?), FP8/FP4 support (how much does quantization buy back?), then FLOP/s. Reading in the reverse order — the marketing order — buys the wrong card.

What Chapter 3 sets up

Once the GPU spec sheet has been re-read through the bandwidth-and-VRAM lens, the natural next question is whether general-purpose GPUs are the right substrate at all. Chapter 4 walks the specialized silicon alternatives — Groq's LPU, AWS Inferentia2, Google's TPU v5p and v6, Intel's Gaudi 3 — each of which is an argument that LLM inference is regular enough to justify a purpose-built chip. Chapter 5 then walks the software-side move that shrinks per-token bandwidth directly: quantization from FP16 down to FP8 and FP4, which turns a bandwidth-bound workload into one whose bottleneck moves.

Next — Chapter 4: Specialized AI Silicon and ASICs. The purpose-built accelerators — Groq, Inferentia2, TPU, Gaudi 3 — and the regimes where they beat GPUs at latency or cost per token.

Want the full picture? The book chapter includes the full spec-comparison table with MSRP and TDP, the arithmetic for tensor parallelism when VRAM ceilings force multi-GPU, the SXM-vs-PCIe distinction that trips up benchmarks, and the In Plain English sidebar on how to read a spec sheet for LLM serving. View LLM Primer VI on Amazon →