Chapter 10 — The LLM Engine Layer

Tenth post of the chapter-by-chapter walkthrough of LLM Primer VI: Scaling AI Systems. The chapter that names the boundary between the engine and the platform, and walks the five engines that dominate that layer in 2026.

Why this chapter exists

Chapters 1–9 walked the machinery a single forward pass touches: the autoregressive loop, the KV cache, the GPU substrate, the quantization that shrinks it, the batching that hides its idle time, and the speculative decoding that breaks its sequential dependency. None of that ships as a library you pip install and forget. Someone wires it into a single-node runtime that wraps a model, owns the KV cache, schedules requests onto a continuous batch, and exposes an inference API. That runtime is the engine. Chapter 10 names the engine layer's job, distinguishes it from the platform layer (Chapter 11), and walks the five engines — vLLM, TensorRT-LLM, SGLang, TGI, Ollama — that make different mechanical trade-offs on the same job.

10.1 vLLM is the Python-native default

vLLM is the engine most production teams should reach for first, because it makes the right default decision on every axis a beginner would not know how to think about, and it does so in Python. PagedAttention gives it fragmentation collapse from 60–80 percent down to single digits, which roughly doubles achievable batch size on the same GPU. Continuous batching layers on top naturally, with chunked prefill mixing prefill and decode work in the same iteration so the boundary between them is not dead time. Copy-on-write prefix sharing is free from the block-table design. The interface is genuinely offline-batch or OpenAI-compatible HTTP in a few lines, and every new model architecture lands there fast because the community is broad. It is the engine to standardize on when the operator does not have a specific reason to pick something else.

10.2 TensorRT-LLM buys throughput with a build pipeline

TensorRT-LLM's pitch is narrow. If the fleet is exclusively NVIDIA, if every percentage point of throughput per dollar matters, and if the team will pay an engineering tax to compile model-specific engine files per hardware generation, TRT-LLM extracts 15–35 percent more throughput than vLLM on the same hardware. The mechanism is at the kernel level: lower the transformer graph into an NVIDIA-specific IR, fuse adjacent kernels (layernorm + matmul + activation into a single launch), select per-shape optimal kernels from a pre-tuned library, produce a serialized engine, and run it under Triton Inference Server. Fusion matters because kernel launch overhead is 5–10 μs each and a naive 70B forward pass dispatches thousands of them per token. The tax is the build pipeline itself — a per-model, per-GPU, per-batch-regime compilation step that most teams underestimate the operational cost of. SGLang is the other specialization: RadixAttention stores the KV cache for every prompt prefix the engine has ever seen in a radix tree, so two requests sharing a k-token prefix share the KV for exactly those k tokens across batches and across time. On agentic workloads with long shared system prompts and short variant suffixes, SGLang delivers 2–6× throughput over vLLM, and its structured-output DSL enforces JSON schemas at the logit level so the output is guaranteed to validate.

10.3 The decision tree runs through workload shape, not headline throughput

The five engines fan out on a small decision tree. Developer laptop or edge with mixed accelerators → Ollama. Production GPU fleet, exclusively NVIDIA, high QPS, throughput ROI justifies build pipeline → TensorRT-LLM. Mixed hardware or frequent model swaps, workload dominated by prefix-heavy structured patterns (agents, tool calls, long shared prompts) → SGLang. Mixed hardware, general chat workload, deep Hugging Face integration → TGI. Everything else → vLLM. The decision is not permanent: engines are interchangeable at the API boundary — they all speak OpenAI-style HTTP — so a platform-layer router can shift traffic per model, per workload, or per region without changing client code. Many production stacks run two or three engines side by side. Beware the headline benchmark: "tokens per second on Llama-2-7B at batch 1" answers a question no production workload asks. Benchmark on your own model, your own prompt distribution, your own concurrency profile; half a day, saves months.

What Chapter 10 sets up

Every engine described in this chapter stops at the same boundary. It knows about kernels, KV blocks, and continuous batching; it does not know about replicas, tenants, chains, or authentication. Chapter 11 walks the platform layer that handles those concerns — Ray Serve, KServe, BentoML, Triton Inference Server — and shows how the choice is much less about features than about which operational model matches the team's existing infrastructure and competencies.

Next — Chapter 11: The Platform and Orchestration Layer. Ray Serve, KServe, BentoML, Triton — the four platforms that sit above the engine, chosen on ops-culture fit.