The KV Cache: The Data Structure That Decides Your Inference Economics

Why the KV cache decides your GPU bill

Every token your model generates has a hidden cost that does not show up on your invoice. It shows up in memory. The KV cache is the data structure that holds it, and once you understand its shape, you stop being surprised by why a 70B model with a long context costs what it costs, why your serving throughput collapses past a certain prompt length, and why every serious inference team in the world is obsessed with attention memory layout.

This is Part 9 of the AI Engineer Series. The previous eight posts were about the engineering around your LLM calls. The next few are about what happens inside the call. We start here because the KV cache is the single largest variable in inference economics that most application teams never look at.

What the KV cache actually is

A transformer decoder, at each generation step, needs to attend over every previous token. Naively, that would mean recomputing the key and value projections for the entire prefix on every step. That is quadratic in sequence length and unworkable.

The fix is to compute K and V once per token and keep them around. The next step reads the cached K and V, computes attention with the new Q, and appends one more K, V pair for the just-generated token. Generation becomes linear in sequence length instead of quadratic. The price is memory. A tensor that grows by one row per token, per attention head, per layer, for the entire lifetime of the request.

The shape is what catches people out. For a single request, the KV cache size is:

kv_bytes = 2 * num_layers * num_kv_heads * head_dim * seq_len * batch * dtype_bytes

The 2 is for K and V. Note num_kv_heads, not num_attention_heads. Modern models use grouped-query attention (GQA) or multi-query attention (MQA), where many query heads share a single KV head. That decision alone cuts KV memory by 4-8x on most production models, and it is why Llama 3 70B is even servable on a single 8-GPU node at long context.

The memory math, with real numbers

Take Llama 3 70B in FP16: 80 layers, 64 query heads, 8 KV heads, head_dim 128. For one request at 8K context:

2 * 80 * 8 * 128 * 8192 * 1 * 2 bytes = ~2.7 GB

That is per request. Push that to 32K context and you are at 10.7 GB. Run a batch of 16 concurrent requests at 32K each and you need 172 GB of KV cache memory alone, before model weights, activations, or anything else. Llama 3 70B weights are 140 GB in FP16. The KV cache for a moderate batch can be larger than the model itself.

This is the single most counterintuitive thing about LLM serving. People assume the model weights dominate. For short prompts and small batches, they do. The moment you push context length or concurrency, the KV cache wins, and it scales linearly in every dimension at once.

Now run the same math without GQA, pretending num_kv_heads equals num_attention_heads at 64. Suddenly that 2.7 GB per request becomes 21.6 GB. The reason GQA is the dominant architecture choice today is not capability. It is serving economics.

Why long contexts get expensive (and slow)

Three things scale with sequence length in a transformer, and they hit at different times.

The prefill phase is compute-bound. You are processing the entire prompt in one forward pass, computing K, V, and attention for every token in parallel. Latency grows roughly linearly with prompt length but the FLOPS are saturated. This is what you wait for at "time to first token."

The decode phase is memory-bound. For each new token, you load the entire KV cache from HBM, multiply against a single Q vector, write back one new K, V pair. The arithmetic intensity is terrible, the GPU spends most of its time waiting on memory bandwidth, and your "time per output token" creeps up as the cache grows. Past 32K tokens, decode is dominated by KV cache reads, not by the matrix multiplies you might expect.

Throughput, finally, is bounded by how many requests you can fit in GPU memory at once. Past a certain context length, the KV cache for one request consumes so much memory that you cannot batch anything else with it. Your tokens-per-second per dollar drops off a cliff, and the cliff is steeper than the pricing curves on the API would suggest.

This is why API providers charge so much more for long-context tokens, why prompt caching (Part 7) has such a dramatic effect on price, and why anyone serving open-weight models obsesses over KV layout. The cache is the bottleneck.

PagedAttention, and why vLLM ate the world

Before vLLM, serving systems allocated KV cache as one contiguous tensor per request, sized for the maximum possible context. That meant a request expecting up to 4096 tokens reserved 4096 tokens of KV memory upfront, even if it only generated 200. Fragmentation and over-allocation routinely wasted 60-80% of cache memory.

PagedAttention, introduced by the vLLM paper, treated KV memory like an operating system treats virtual memory. The cache is split into fixed-size blocks (typically 16 tokens each). A logical-to-physical block table maps a request's "sequence of blocks" to wherever those blocks actually live in GPU memory. New blocks are allocated on demand as generation proceeds. Free blocks return to a pool.

The result is near-zero fragmentation and 2-4x higher serving throughput on the same hardware, with no quality change. Almost every modern serving stack (vLLM, SGLang, TensorRT-LLM) uses some variant of paged KV management today. If you are running your own inference and you are not, you are leaving half your hardware on the table.

Prefix sharing: where the real wins are

The other thing paging unlocked was sharing. If two requests have the same prefix (system prompt, few-shot examples, retrieved context), their KV caches for that prefix are bit-identical. Why store it twice?

Paged systems can reference-count blocks. The system prompt's KV cache is computed once, stored once, and referenced by every request that includes it. RadixAttention (SGLang) goes further: it maintains a radix tree of cached prefixes and matches incoming requests against it. Common prefixes are reused; only the divergent suffixes are computed and stored.

In practice, on a workload where 80% of every prompt is a shared system prompt and templates, prefix sharing alone can deliver 3-5x throughput improvement. This is the server-side equivalent of the prompt caching you saw in Part 7 from the API provider's perspective. Inside their inference cluster, what you experience as "cached input tokens at a discount" is in part the result of radix-tree KV sharing across requests in the same tenant.

What you actually need to do

Most application teams will never write a kernel that touches the KV cache. They still need to make decisions that depend on understanding it. Three concrete moves:

Choose models with GQA or MQA for production serving. If you are picking between two otherwise-comparable open-weight models, the one with fewer KV heads will serve at roughly 4-8x lower KV memory cost per request. This shows up directly in your max batch size and your cost per token.

Treat context length as a serving cost, not a free parameter. A 32K-token prompt is not 4x the cost of an 8K prompt. It is 4x the KV memory, and once batch sizes shrink to accommodate it, your effective cost per token rises faster than linearly. Trim aggressively. Most "we need long context" requirements are actually retrieval problems in disguise, which is what Part 2 was about.

Structure prompts to maximize prefix sharing. Put the stable system content and shared examples first. Put the variable, request-specific content last. This is exactly the same advice that makes prompt caching work, because under the hood it is the same mechanism. Stable prefix in front, variable suffix in back.

What is next

Part 10 covers the two other inference-side levers that compose with KV cache management: speculative decoding (EAGLE-3, Medusa) and quantization (INT8, INT4, AWQ). They change different parts of the bottleneck and they stack with everything in this post. Together they decide whether your self-hosted inference is competitive with the API or just expensive.

Previous in the series

• Part 1: Harness Engineering
• Part 2: Context Engineering & Retrieval Quality
• Part 3: Structured Outputs and Fallback Chains
• Part 4: Evals
• Part 5: LLM Observability
• Part 6: Latency and Throughput Engineering
• Part 7: Prompt Caching vs Semantic Caching
• Part 8: Cost Attribution Per Feature

References

1. Efficient Memory Management for Large Language Model Serving with PagedAttention by Kwon et al. The vLLM paper. Foundational for modern KV cache management.
2. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints by Ainslie et al. The paper behind why your favorite open-weight model has fewer KV heads than query heads.
3. SGLang: Efficient Execution of Structured Language Model Programs by Zheng et al. Introduces RadixAttention for prefix sharing across requests.
4. vLLM PagedAttention design documentation. The implementer's view of how blocks, block tables, and the allocator actually work.
5. Transformer Inference Arithmetic by Kipply Chen. The clearest walkthrough of where compute and memory go in transformer inference, with the math worked out end to end.