Chapter 5 — Demystifying Quantization

Fifth post of the chapter-by-chapter walkthrough of LLM Primer VI: Scaling AI Systems. The chapter that explains why a 70B model survives 4-bit quantization while a 1B model does not — and how to pick the recipe.

Why this chapter exists

Decoding is memory-bandwidth-bound; the currency of bandwidth is bytes-per-weight. A 70B model in BF16 reads 140 GB per forward pass. The same model in INT4 reads 35 GB. The arithmetic is identical. The bottleneck moves by a factor of four. That single observation is why quantization has gone from a research curiosity to the default deployment path for production inference. Chapter 5 walks the mechanics — what precision actually means, why aggressive quantization does not destroy a large model the way it destroys a small one, what AWQ, GPTQ, SmoothQuant, and GGUF are actually doing under the hood, and where quantization stops being safe and starts silently degrading quality.

5.1 Why big models survive 4-bit and small ones do not

Three observations underwrite large-model quantization. First, the information content of an individual weight in a large transformer is low: predictive behavior emerges from the collective interaction of billions of weights whose distribution is sharply peaked near zero. Rounding a weight from 0.0031 to 0.003 changes nothing the next layer can detect. Second, trained transformers sit inside a flat region of parameter space where many nearby configurations produce nearly identical outputs, and for models above 30B or so the region is wide enough to absorb 4-bit perturbation with sub-point MMLU degradation. Third, mixed precision lets sensitive layers — attention scores, layer norms, the final logits — stay in BF16 while the bulk linear projections drop to INT4. A 70B model in INT4 is empirically nearly indistinguishable from its BF16 self; a 1B model in INT4 is noticeably worse, because the flat region is smaller and the perturbation pushes the model out of it. The naive intuition — that smaller models should quantize more easily — is exactly backwards.

5.2 AWQ, GPTQ, SmoothQuant, and GGUF do different things

GPTQ walks the weight matrix column by column, picks quantization levels to minimize output error against a small calibration set, and updates the still-unquantized columns to absorb the residual — a Hessian-approximated compensation that keeps the layer's output close to original. AWQ starts from a different observation: activation outliers matter as much as weight outliers, so scale up the salient weight channels before quantizing (and scale the activations down to compensate), which spreads the quantization levels over the range those weights actually occupy. SmoothQuant attacks the activation side: LLMs have a handful of channels with enormous magnitudes that wreck naive activation quantization, so it migrates the outlier magnitude from activations into weights per-channel, letting W8A8 land with negligible loss. GGUF is a file format rather than a single algorithm — the Q4_K_M-style nested super-block scaling used by llama.cpp for CPU and edge inference, ecologically important but rarely used on data-center GPUs.

5.3 The safety ladder and the calibration discipline

The empirical safety ladder is clean. BF16 → FP8 is nearly always lossless and is the default for production. BF16 → INT8 is lossless above ~7B with a competent algorithm. BF16 → INT4 is lossless above ~30B with AWQ or GPTQ; below 13B it costs one to three MMLU points; below 7B it costs five or more without quantization-aware training. INT3 and below is experimental. Two disciplines govern whether the ladder holds in production. The first is calibration on the right distribution: 128–512 representative samples drawn from the same distribution the production workload will draw from, re-run every six to twelve months as the workload drifts. The second is task-side evaluation: standard benchmarks can miss quality shifts on the long tail of the model's capability — rare facts, multi-step reasoning, minority-language code — and aggressive quantization should be validated against a slice of real production traffic scored by the dimensions that matter for the application.

What Chapter 5 sets up

Quantization shrinks bytes per weight. The next chapter shrinks the count of weights directly. Chapter 6 walks pruning — including the 2:4 structured sparsity that Hopper accelerates natively — and knowledge distillation, where a large teacher's behavior is transferred into a smaller student that runs cheaper end to end. Together, the three compressions (quantize, prune, distill) form the model-side toolkit for lowering the bandwidth burden that Chapter 1 named. Chapter 7 then turns to the runtime-side lever — batching — that converts the newfound headroom into concurrent user throughput.

Next — Chapter 6: Pruning and Knowledge Distillation. The two model-side compressions that attack weight count directly rather than weight width.