Chapter 14 — Token Economics and API Pricing

Fourteenth post of the chapter-by-chapter walkthrough of LLM Primer VI: Scaling AI Systems. The chapter that connects the physics of Chapter 1 to the line items on the invoice — and explains why the first month's bill often bears no resemblance to what the team modeled.

Why this chapter exists

Earlier chapters treated the cost of inference as a physical fact — bytes moved across HBM, FLOPs spent in a matmul, GPU-seconds consumed by a request. That framing is correct for the team that owns the silicon. For the much larger population of teams that buy inference from an API, the cost surface is reshaped by the unit the provider charges for: the token. Chapter 14 is about why the token is priced the way it is, why the input and output sides of the same call have such different prices, and how quietly a chatbot's conversation history and a reasoning model's invisible thinking end up on the invoice.

14.1 Output costs 4–8× input because decoding is the bandwidth-bound phase

An input token rides the prefill path — dense, compute-bound, parallel across all positions of the prompt, riding exactly the workload GPUs are designed for. On a 70B model an H100 chews through tens of thousands of input tokens per second because the operation is a wide matmul with high arithmetic intensity. An output token rides the decode path — sequential, memory-bandwidth-bound, one token per forward pass, each step reading the entire KV cache from HBM. The same H100 produces 50–100 output tokens per second per request. The ratio between the two throughputs is the ratio the price sheet reflects. Frontier models in 2025 charge two to five dollars per million input tokens and ten to thirty dollars per million output — a 4:1 to 8:1 asymmetry that is not a margin choice but a direct pass-through of the underlying compute asymmetry. Cached input is cheaper still because the provider paid the prefill cost on a previous request. Batch APIs are half-price because the provider gets to schedule the work into idle capacity windows. Reasoning tokens are output tokens because that is what they are inside the engine.

14.2 The optimization lever depends on the input/output shape

The mistake is to model cost with one mental number — "tokens cost X" — instead of decomposing input and output separately. A retrieval-heavy customer-service triage sending 8,500 input tokens and getting a 600-token response is dominated by input cost at frontier pricing (roughly $255 input to $90 output per 10,000 requests per day). A long-form drafting system on the same volume — 2,200 input tokens in, 3,000 output tokens out — is dominated by output ($66 input to $450 output). The optimization advice is opposite in the two cases: shorten the prompt for the first, shorten the response for the second, and route to a model whose pricing profile favors the dominant axis. A second decomposition worth running is by tenant or workload. Multi-tenant systems almost always show a power-law skew where the top 10 percent of tenants account for half or more of spend, and cost-optimization work targeted at those tenants pays back several times more than uniform work spread across the long tail.

14.3 Context accumulation and invisible reasoning tokens compound the invoice

The most expensive token is the one the team forgot was being sent. APIs are stateless; every request ships the full conversation history as input on every call. The cumulative input cost across an N-turn conversation grows quadratically in N, minus prefix-cache credit. A team that priced its system on a single-turn baseline finds, six months later, that its average conversation now spans eight turns and its per-conversation input cost has multiplied roughly as the square. The second silent multiplier is reasoning tokens. Models like o1/o3, Anthropic's extended thinking, DeepSeek R1, and Google's reasoning variants produce a large volume of internal thinking before the visible answer — often 4,000–8,000 tokens behind a 500-token visible response. Those tokens ride the same output decode loop and are billed as output; they are not returned to the caller unless the code asks for them. The mitigation is to instrument the API response's usage object correctly — prompt_tokens, cached_prompt_tokens, completion_tokens, reasoning_tokens, and any provider-specific categories — and to flag unknown categories rather than silently dropping them. Every new billed dimension the provider adds must show up in the team's usage schema, or the next invoice is a surprise.

What Chapter 14 sets up

Once the team can read its own invoice with the input-versus-output, per-tenant, and per-category decomposition, the next question — the one finance will ask — is whether the team should be paying per token at all. Chapter 15 walks the alternative: bring inference in-house, rent or buy GPUs, and pay per GPU-hour instead. The break-even math is clean. What is less clean, and what most teams underestimate, is everything on the dedicated side of the ledger that is not on the GPU invoice: the platform engineering, the security patches, the model upgrade choreography, the on-call rotation that answers when a node falls out of the fleet.

Next — Chapter 15: Serverless APIs vs Dedicated Infrastructure. The break-even formula, the hidden line item, and hybrid postures where both is the right answer.