Chapter 4 — Specialized AI Silicon and ASICs
Fourth post of the chapter-by-chapter walkthrough of LLM Primer VI: Scaling AI Systems. The chapter that argues the choice between a GPU and an ASIC is a workload-shape question — and gives you the shape.
Why this chapter exists
The GPU camp argues that LLM inference is one workload in a broad class of dense matmul problems and that a programmable accelerator is the right substrate. The ASIC camp argues that inference is specific, regular, and forecast-able enough that a purpose-built chip can beat a general one on the two metrics that matter — latency and cost per token. Both camps are right about the workloads they choose. Chapter 4 walks the production-grade ASIC alternatives to data-center GPUs in 2026 — Groq's LPU, AWS Inferentia2, Google TPU v5p and v6, Intel Gaudi 3 — and names the regimes where each wins and the failure modes that bring engineers back to GPUs.
| Silicon | Architecture | Software stack | Wins on | Loses on |
|---|---|---|---|---|
| Groq LPU | No HBM — on-chip SRAM only, deterministic TSP interconnect, rack-scale unit | GroqCloud API, compiler-scheduled | Ultra-low latency — 200–500 tok/s single-user on 70B | Model size (needs 250+ chips for 70B); model variety |
| AWS Inferentia2 | Neuron cores, HBM, systolic engine | Neuron SDK + vLLM container; compile-then-cache | 40–60% $/M tok savings on stable AWS workloads | Frequent model swaps (compile tax); non-AWS deploys |
| Google TPU v5p / v6 | Systolic MXU, HBM, ICI interconnect | JAX/XLA via Vertex AI | Steady-state batched serving; Google-scale training reuse | Kernel research (XLA constraints); GCP-only |
| Intel Gaudi 3 | Matrix + tensor cores, HBM, 200/400 GbE for scale-out | SynapseAI, PyTorch Lightning, growing vLLM | Cost lever w/ least migration friction; Ethernet-native fabric | Ecosystem depth vs CUDA |
| NVIDIA GPU (baseline) | Programmable SMs, HBM, NVLink | CUDA + full engine ecosystem | Model variety, frontier scale, custom kernels | Purpose-built cost ceiling on steady workloads |
4.1 Groq eliminates HBM to make latency deterministic
The Groq LPU is the most architecturally distinctive chip in the production accelerator market: it has no HBM. The entire model lives in compiler-scheduled on-chip SRAM, distributed across many chips connected by a deterministic on-package network. There is no dynamic memory allocation, no kernel-launch overhead, no scheduling jitter — every run of the same input produces the same output in exactly the same number of nanoseconds. The metric that falls out is decoding latency GPUs cannot approach: a 70B model at 200–500 tokens per second per request, against 30–80 on H100. The architectural cost is model size — SRAM is bounded to hundreds of megabytes per chip, so a 70B model requires a rack of 250+ chips connected by Groq's TSP interconnect, and the rack is the unit of deployment. The workloads that pay for that rack are voice assistants under a 200 ms conversational budget, interactive code completion, financial signal generation — anywhere the user perceives every millisecond.
4.2 Inferentia2 and TPU win on cost when the model is stable
AWS Inferentia2 and Google TPU v5p/v6 attack a different metric: dollars per million output tokens. Both are purpose-built for steady-state batched serving where the model portfolio is narrow, the latency budget is loose, and the volume is high enough that a 40–60 percent cost saving over GPUs dominates the re-platforming cost. Inferentia2 exposes the Neuron SDK and runs vLLM through the official AWS container; the compilation step is heavy on first load and cached thereafter. TPU exposes JAX/XLA and runs through Vertex AI Inference endpoints. Both platforms reward workloads that look the same day after day — translation, moderation, embedding pipelines, RAG over stable corpora — and both punish workloads that change model architecture frequently, because the compile-then-cache path is a tax on every model swap. The discipline is to pick one and standardize; running both at production scale is a re-platforming tax doubled.
4.3 Gaudi 3 is the Ethernet-first bet, and GPUs still win the frontier
Intel Gaudi 3 makes a different structural bet: rather than a proprietary interconnect (NVLink, ICI, TSP), it uses commodity 200/400 GbE for chip-to-chip communication, betting that Ethernet-native scale-out lowers the operational cost of large deployments. The software is Habana's SynapseAI plus PyTorch Lightning integrations, with steadily improving vLLM support. Gaudi 3 is priced aggressively against H100 and MI300X and lands as the "cost lever with the least migration friction" for teams that want ASIC economics without a new interconnect layer. Against all of these, GPUs still win in one regime: model variety, frontier scale, and experimentation. Anywhere the team runs a 405B-class model, a freshly released frontier architecture, a custom-fine-tuned variant, or a research-paper kernel, CUDA's ecosystem depth and NVIDIA's first-to-market position pay back. The heuristic is the "kernel question" — if the stack runs standard FlashAttention and standard fused-MLP kernels and never touches anything else, an ASIC is a candidate; if the stack runs a kernel that someone on the team wrote, a GPU is the safer buy.
What Chapter 4 sets up
Chapters 3 and 4 have described the hardware substrate as it stands. Every card on offer is either constrained by HBM bandwidth (GPU decoding) or has re-engineered away from it (Groq) or has priced its bandwidth aggressively for stable workloads (Inferentia, TPU, Gaudi). Chapter 5 turns to the software-side move that shrinks the per-token bandwidth burden directly: quantization from FP16 down to FP8 and FP4. The gain from a Blackwell upgrade and the gain from quantizing an existing model to FP8 are, mechanistically, the same lever pulled at different layers of the stack — and both are usually applied together.
