Chapter 11 — The Platform and Orchestration Layer

Eleventh post of the chapter-by-chapter walkthrough of LLM Primer VI: Scaling AI Systems. The chapter that argues the platform choice is not about features but about which operational model matches the team's existing ops culture.

Why this chapter exists

An engine wraps a model on a GPU and produces tokens. That is its complete job description. The moment production needs two replicas behind a load balancer, or two different engines under one endpoint, or a pipeline that runs embed→retrieve→rerank→generate as one logical call, or a quota system that stops one tenant starving another, or an autoscaler that adds replicas as queue depth rises, the engine has run out of answers. Chapter 11 walks the four platforms that dominate that job in 2026 — Ray Serve, KServe, BentoML, Triton Inference Server — and argues the choice between them is a fit question about the team's ops culture, not a features question.

11.1 Ray Serve is Python-first; KServe is Kubernetes-first

Ray Serve is the platform that emerges when the starting question is "what would a serving framework look like if I never had to leave Python to compose, scale, and deploy a model pipeline?" The answer is Python classes decorated as actors, wired together with Python function calls, replicated and placed and autoscaled by the Ray runtime. Deployments compose into pipelines by calling each other's methods; the pipeline is Python, the observability is Python, the ops story is Python. It is the natural choice when the same cluster is already running Ray for training. KServe emerges from the opposite starting point: Kubernetes-first, vendor-neutral, declarative. A KServe InferenceService is a CRD that captures where the weights live, what runtime serves them, and how autoscaling should behave. The platform wires the pods through Knative for scale-to-zero and Istio for traffic routing and exposes a stable HTTP endpoint. The trade is expressiveness — pipeline composition in KServe is more YAML-driven than in Ray Serve — for portability and standardization.

11.2 BentoML packages; Triton batches heterogeneous models

BentoML takes a third stance: the platform's primary job is to make model + code + dependencies a single coherent artifact — a Bento — and to make deploying that artifact to anywhere (Kubernetes, Lambda, Cloud Run, a bare VM) a one-command operation. It is what a Docker image is to general applications, at the model-serving layer. The authoring experience is FastAPI-shaped Python; the deploy story is the artifact-first move. Best fit is a small ops footprint and a team that wants the fastest path from "Python class" to "deployed endpoint that runs anywhere." Triton Inference Server is the enterprise-grade dynamic batcher that historically dominated NVIDIA-aligned serving. It hosts TensorFlow, PyTorch, ONNX, TensorRT, TensorRT-LLM, and Python backends side by side, applies its dynamic batcher to non-LLM workloads (recommenders, classifiers, embedders), and uses the continuous-batching scheduler for LLMs under its TensorRT-LLM backend. Best fit is heterogeneous workloads or a team already deep in the NVIDIA stack.

11.3 The four choices partition on ops-culture fit

The four platforms partition cleanly. Python and Ray as center of gravity → Ray Serve. Kubernetes, declarative manifests, multi-cloud portability, multi-tenant isolation → KServe. Small ops footprint, fastest path from class to endpoint → BentoML. Heterogeneous workloads or NVIDIA-first shop → Triton. They are not mutually exclusive at the organizational level: a common pattern in larger orgs is Triton for the highest-volume production LLMs, Ray Serve for the RAG and agent pipelines that compose multiple model calls, KServe as the umbrella that enforces multi-tenancy, and BentoML inside developer environments. The useful exercise when choosing is to list three operational properties the team absolutely needs and three it can absolutely live without, then rank-order the platforms against that list. Resist the platform that "supports everything" — each platform supports its core well and its periphery poorly, and the one whose core overlaps the must-haves is the one the team will enjoy operating.

What Chapter 11 sets up

The engine/platform boundary is clean when the unit of replication is one pod owning one model on one GPU node. It stops being clean the moment we honor Chapter 1's fact that prefill and decode are contradictory workloads. Chapter 12 splits them onto separate GPU pools, ships the KV cache between the pools over the network fabric, and expresses the split in Kubernetes primitives — LeaderWorkerSet, Grove's PodCliqueSet, KAI Scheduler's topology awareness. The platform layer is what makes the split operable; the platform layer is why the split is expressible at all.

Next — Chapter 12: Disaggregated Serving and Kubernetes. Splitting prefill and decode into separate pods and the Kubernetes primitives that express the split.