Chapter 4 — Specialized AI Silicon and ASICs

Fourth post of the chapter-by-chapter walkthrough of LLM Primer VI: Scaling AI Systems. The chapter that argues the choice between a GPU and an ASIC is a workload-shape question — and gives you the shape.

Why this chapter exists

The GPU camp argues that LLM inference is one workload in a broad class of dense matmul problems and that a programmable accelerator is the right substrate. The ASIC camp argues that inference is specific, regular, and forecast-able enough that a purpose-built chip can beat a general one on the two metrics that matter — latency and cost per token. Both camps are right about the workloads they choose. Chapter 4 walks the production-grade ASIC alternatives to data-center GPUs in 2026 — Groq's LPU, AWS Inferentia2, Google TPU v5p and v6, Intel Gaudi 3 — and names the regimes where each wins and the failure modes that bring engineers back to GPUs.

One line: ASICs beat GPUs when the workload is either latency-critical or cost-sensitive and the model doesn't change often — the kernel question and the model-change-frequency question decide the rest.

4.1 Groq eliminates HBM to make latency deterministic

The Groq LPU is the most architecturally distinctive chip in the production accelerator market: it has no HBM. The entire model lives in compiler-scheduled on-chip SRAM, distributed across many chips connected by a deterministic on-package network. There is no dynamic memory allocation, no kernel-launch overhead, no scheduling jitter — every run of the same input produces the same output in exactly the same number of nanoseconds. The metric that falls out is decoding latency GPUs cannot approach: a 70B model at 200–500 tokens per second per request, against 30–80 on H100. The architectural cost is model size — SRAM is bounded to hundreds of megabytes per chip, so a 70B model requires a rack of 250+ chips connected by Groq's TSP interconnect, and the rack is the unit of deployment. The workloads that pay for that rack are voice assistants under a 200 ms conversational budget, interactive code completion, financial signal generation — anywhere the user perceives every millisecond.

4.2 Inferentia2 and TPU win on cost when the model is stable

AWS Inferentia2 and Google TPU v5p/v6 attack a different metric: dollars per million output tokens. Both are purpose-built for steady-state batched serving where the model portfolio is narrow, the latency budget is loose, and the volume is high enough that a 40–60 percent cost saving over GPUs dominates the re-platforming cost. Inferentia2 exposes the Neuron SDK and runs vLLM through the official AWS container; the compilation step is heavy on first load and cached thereafter. TPU exposes JAX/XLA and runs through Vertex AI Inference endpoints. Both platforms reward workloads that look the same day after day — translation, moderation, embedding pipelines, RAG over stable corpora — and both punish workloads that change model architecture frequently, because the compile-then-cache path is a tax on every model swap. The discipline is to pick one and standardize; running both at production scale is a re-platforming tax doubled.

4.3 Gaudi 3 is the Ethernet-first bet, and GPUs still win the frontier

Intel Gaudi 3 makes a different structural bet: rather than a proprietary interconnect (NVLink, ICI, TSP), it uses commodity 200/400 GbE for chip-to-chip communication, betting that Ethernet-native scale-out lowers the operational cost of large deployments. The software is Habana's SynapseAI plus PyTorch Lightning integrations, with steadily improving vLLM support. Gaudi 3 is priced aggressively against H100 and MI300X and lands as the "cost lever with the least migration friction" for teams that want ASIC economics without a new interconnect layer. Against all of these, GPUs still win in one regime: model variety, frontier scale, and experimentation. Anywhere the team runs a 405B-class model, a freshly released frontier architecture, a custom-fine-tuned variant, or a research-paper kernel, CUDA's ecosystem depth and NVIDIA's first-to-market position pay back. The heuristic is the "kernel question" — if the stack runs standard FlashAttention and standard fused-MLP kernels and never touches anything else, an ASIC is a candidate; if the stack runs a kernel that someone on the team wrote, a GPU is the safer buy.

Worth holding onto: The three workload archetypes rarely shade into each other. Ultra-low-latency real-time → Groq. Steady-state high-volume → Inferentia or TPU. Model variety at the frontier → GPUs. The expensive mistakes come from being in one category and shopping in another.

What Chapter 4 sets up

Chapters 3 and 4 have described the hardware substrate as it stands. Every card on offer is either constrained by HBM bandwidth (GPU decoding) or has re-engineered away from it (Groq) or has priced its bandwidth aggressively for stable workloads (Inferentia, TPU, Gaudi). Chapter 5 turns to the software-side move that shrinks the per-token bandwidth burden directly: quantization from FP16 down to FP8 and FP4. The gain from a Blackwell upgrade and the gain from quantizing an existing model to FP8 are, mechanistically, the same lever pulled at different layers of the stack — and both are usually applied together.

Next — Chapter 5: Demystifying Quantization. The precision drop from BF16 to FP8 to INT4, why big models survive it, and the calibration workflow that actually ships.

Want the full picture? The book chapter includes the runnable Groq, Inferentia2, TPU, and Gaudi call examples, the Neuron cache mechanics, the JAX-on-TPU compilation story, and the In Plain English sidebar on reading a workload to pick silicon. View LLM Primer VI on Amazon →