Speculative Decoding and Quantization: The Two Levers That Compound

Two levers that change different bottlenecks

Part 9 showed why the KV cache decides your serving economics. This post covers the two other techniques that move the needle on inference cost and latency: speculative decoding and quantization. They work on different parts of the bottleneck, they stack with each other, and they stack with paged KV management. Together they are the difference between self-hosted inference that competes with the API and self-hosted inference that just looks cheap on paper.

This is Part 10 of the AI Engineer Series. We will look at what each technique actually does, where the real numbers are today, and when to compose them.

Why decoding is slow in the first place

Recall from Part 9 that decoding is memory-bound. To generate one token, the GPU loads the entire KV cache and the model weights from HBM, does a tiny amount of arithmetic, and writes back one new K, V pair. The arithmetic intensity is so low that the matrix multiplication units sit mostly idle waiting on memory bandwidth. A modern H100 has 80 GB of HBM at 3.35 TB/s. For a 70B model in FP16, that is roughly 24 reads of weights per second, capping you near 24 tokens/sec per request before you have done anything clever.

Speculative decoding and quantization both attack this in different ways. Speculative decoding generates multiple tokens per HBM read by guessing them and verifying in parallel. Quantization shrinks the weights so each HBM read carries more parameters per byte.

Speculative decoding, the core idea

You have an expensive target model. You also have a cheap draft model (or a smaller head, or a parallel branch) that produces token guesses fast. The draft model proposes K tokens speculatively. The target model verifies them in a single forward pass that costs roughly the same as generating one token, because the bottleneck is HBM bandwidth, not compute.

If the draft model's guesses agree with the target model on the first j tokens, you accept those j tokens and pay roughly the cost of one target-model step. The math works out to a 2-3x speedup in most settings, with no quality change because the target model verifies everything. The losses come when the draft model guesses wrong and tokens get rejected.

The original 2023 paper from DeepMind and a parallel paper from Google introduced this with a draft model of the same architecture but smaller (a 7B drafting for a 70B). Modern variants do much better.

Medusa, EAGLE, and the heads vs. drafter tradeoff

Medusa skipped the separate draft model. Instead, it attaches multiple "decoding heads" to the target model that each predict tokens at different offsets. Heads are fine-tuned, the rest of the model is frozen. Verification uses a tree-shaped attention pattern that checks multiple draft sequences at once. Throughput improvements were around 2-3x on Vicuna 7B and 13B at the time of the paper, with no draft model to load or train separately.

EAGLE went further. EAGLE-2 and EAGLE-3 use a small auto-regressive head trained on the target model's hidden states (not just the logits), and a dynamic draft tree that adapts to how confident the head is. EAGLE-3 in particular reports speedups of around 3-5x on Llama 3 and Qwen, with quality bit-identical to the target.

What this means in practice: if you are serving a frozen model and you can fine-tune a small head, EAGLE-style speculative decoding gets you most of the speedup with minimal infrastructure complexity. If you do not control training, a separate small draft model from the same family still works and is supported by most modern inference servers today.

Quantization, what actually gets smaller

Quantization reduces the numerical precision of the weights, and sometimes the activations or the KV cache, from FP16 to INT8 or INT4. The savings show up in three places: HBM footprint (lets you fit bigger models on the same GPU), HBM bandwidth (each read carries more parameters), and sometimes compute (INT8 matmuls run faster on tensor cores).

The naive approach is round-to-nearest with a per-tensor scale. This works for INT8 with very small quality loss on most models. INT4 is where it gets hard. At 4 bits per weight, you are cutting precision so aggressively that the quantization error starts to compound through layers and the model degrades visibly.

The two techniques that made INT4 production-viable on dense LLMs are GPTQ and AWQ.

GPTQ uses second-order information (a Hessian approximation) to quantize weights one column at a time, while compensating the remaining columns for the rounding error introduced so far. It is post-training, takes a few hours on a calibration dataset, and reaches INT4 with around 0.1-0.5 perplexity degradation on Llama-class models.

AWQ (Activation-aware Weight Quantization) observed that not all weights are equally important. About 1% of weight channels handle most of the salient activations. AWQ scales these "important" channels up before quantizing, which preserves precision where it matters and accepts more loss where it does not. AWQ generally beats GPTQ slightly at INT4 and is faster to apply.

For most production serving today, AWQ at INT4 (or INT4 weights + FP16 activations, "W4A16") is the sweet spot for open-weight models. You get roughly 4x reduction in weight memory and around 2-3x decode speedup with quality loss small enough to be invisible in evals on most tasks.

KV cache quantization is the other half

From Part 9, the KV cache can dominate memory at long context. INT8 or INT4 quantization of the KV cache itself is increasingly common. vLLM, TensorRT-LLM, and llama.cpp all support it. The quality impact is small for INT8 (essentially free), modestly larger but still acceptable for INT4 on most tasks.

Quantizing both weights and KV cache compounds. A Llama 3 70B with W4A16 weights and INT8 KV cache fits comfortably on a single H100 at 32K context with reasonable batch size. In FP16 with no KV quantization, it does not.

When the techniques compose, and when they fight

Speculative decoding and quantization stack cleanly. Quantization speeds up every forward pass, including the target model's verification pass in speculative decoding. Speculative decoding generates more tokens per forward pass. Combined, you see multiplicative speedups on most workloads. EAGLE-3 on top of an AWQ-quantized model is a common production setup.

Paged KV management (Part 9) composes with both. Speculative decoding requires KV cache management that handles multiple candidate continuations during verification; modern serving stacks (vLLM, SGLang, TensorRT-LLM) all support this.

Where the techniques fight: speculative decoding gives less speedup when the draft model's guesses are frequently wrong. On highly specialized or out-of-distribution prompts, the acceptance rate drops and the overhead of running the draft model can actually slow things down. Always measure acceptance rate on your real workload before declaring victory.

Quantization has a cleaner failure mode but a sneakier one. The aggregate evals look fine, then you find a specific task (often structured generation, code, or non-English text) where INT4 has degraded the model noticeably. Evals at the task level (Part 4) are how you catch this. Do not trust a single perplexity number.

The numbers that actually matter

For a Llama 3 70B served with vLLM on a single H100:

• FP16 baseline: roughly 25 tokens/sec/request, batch ~4 at 8K context
• + AWQ INT4 weights: roughly 50-70 tokens/sec/request, batch ~12-16
• + INT8 KV cache: same throughput, batch grows further (often 2x more concurrent requests)
• + EAGLE-3 speculative decoding: roughly 2-3x further on tokens/sec/request

The composition is what gets you from 25 tok/s/req to 150+ tok/s/req on the same hardware, without changing the served model's quality in any user-visible way. Cite these numbers cautiously, because they move with every release and your workload will differ, but the relative order is stable: paged KV is the foundation, quantization is the biggest single multiplier on cost, speculative decoding is the biggest single multiplier on latency.

When to bother

If you are using a hosted API, you mostly do not bother. The provider is doing all of this for you, and your levers are at the prompt and routing layer. Where this matters directly is when you self-host (open-weight models, on-prem, regulated environments) or when you are building a serving product.

The minimum bar for self-hosted serving today is vLLM or SGLang with AWQ-quantized weights and INT8 KV cache. Without those, you are paying 3-5x more in hardware than you need to. Speculative decoding is the next step up, and it is worth the complexity once you have stabilized everything else.

If you are running Llama 3 70B in FP16 with no quantization and a contiguous KV cache, you are roughly where the field was in mid-2023. The community has moved a long way since.

What is next

Part 11 zooms back out to the application layer. We will look at model routing, cascading cheap-to-expensive, classifier-based dispatch, provider failover, and RouteLLM. The inference-side techniques from Posts 9 and 10 decide cost per token; routing decides which model you spend those tokens on in the first place.

Previous in the series

• Part 1: Harness Engineering
• Part 2: Context Engineering & Retrieval Quality
• Part 3: Structured Outputs and Fallback Chains
• Part 4: Evals
• Part 5: LLM Observability
• Part 6: Latency and Throughput Engineering
• Part 7: Prompt Caching vs Semantic Caching
• Part 8: Cost Attribution Per Feature
• Part 9: The KV Cache

References

1. Fast Inference from Transformers via Speculative Decoding by Leviathan et al. The DeepMind paper that introduced the technique.
2. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads by Cai et al. The "no separate draft model" approach.
3. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test by Li et al. The current state of the art in speculative decoding heads.
4. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration by Lin et al. The technique behind most production INT4 deployments today.
5. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers by Frantar et al. The other major INT4 quantization method, slightly older and still widely used.
6. vLLM quantization documentation. Practical guide to which methods are supported and how to enable them.