Latency and Throughput Engineering: The Numbers That Decide if Your AI Feels Alive

"It is slow" is not a problem statement

Every AI engineer eventually gets the same complaint. The product feels slow. The bug ticket says "the AI is laggy." Someone in Slack asks why the chat takes so long.

The honest answer is that "slow" has structure. LLM latency is not one number. Until you can name which part is slow, you cannot fix it. This is Part 6 of the AI Engineer Series, and it covers the three numbers that decide whether your AI feels alive or sluggish.

The three latency numbers that matter

Stop using a single latency number. Track these three separately.

TTFT (Time to First Token). How long from request start until the first token streams back. This is what determines whether your UI feels responsive. Below 500ms feels instant. 500ms to 1s feels prompt. 1s to 2s feels noticeable. Above 2s feels broken, even if the full response arrives quickly afterward.

ITL (Inter-Token Latency). Time between consecutive tokens once streaming starts. Determines how fast text appears on screen. 50ms per token feels comfortable, slightly faster than typical reading speed. 100ms per token feels sluggish. Above 200ms per token, users notice the stutter.

End-to-end. Total request duration from send to last token. The number that matters for non-streaming use cases: background jobs, structured output extraction, eval pipelines, anything where the user is not watching the stream.

For chat and assistant UIs, TTFT dominates perceived speed. For batch processing, end-to-end is what counts. ITL matters when output is long.

Always report percentiles, never averages

The average latency is a number that hides everything important. The p99 is the truth. Report at least p50, p95, and p99 for every metric above.

The ratio matters. If p99 is more than 3x p50, you have tail-latency problems that need investigation. Causes are usually queueing under load, retries firing on a struggling provider, or a small subset of requests with much longer inputs.

import time

def measure_streaming_call(client, model, prompt):
    """Capture TTFT and total duration for a streaming call."""
    start = time.time()
    first_token_time = None
    token_times = []
    
    with client.messages.stream(
        model=model,
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}],
    ) as stream:
        for event in stream:
            if event.type == "content_block_delta":
                now = time.time()
                if first_token_time is None:
                    first_token_time = now
                token_times.append(now)
    
    total = time.time() - start
    ttft = (first_token_time - start) if first_token_time else total
    
    # ITL: median time between consecutive tokens
    deltas = [t2 - t1 for t1, t2 in zip(token_times, token_times[1:])]
    deltas.sort()
    itl = deltas[len(deltas) // 2] if deltas else 0.0
    
    return {"ttft": ttft, "itl": itl, "total": total}

Streaming is the cheapest UX win you have

Streaming does not make the model faster. It makes the user feel like the model is faster, which is what matters. The full response takes the same total time, but the user sees text appearing within a few hundred milliseconds instead of waiting four seconds for the whole thing.

If your chat UI is not streaming, fix that before anything else on this list. It is one afternoon of work and the biggest perceived-latency improvement you will ever ship.

Streaming has two gotchas worth knowing. First, structured outputs cannot be safely rendered until parsing completes, so you stream a "thinking" indicator and reveal the final object when the model is done. Second, tool calls cannot be executed until the full call object streams, so treat tool calls as atomic and use streaming for the visible response.

Where the time actually goes

For a hosted API call, your latency budget looks roughly like this:

• Network: 50-200ms round trip, depending on region. Use the closest provider region.
• Queue time: 0 to several seconds, depending on provider load. Cannot be observed directly, only inferred.
• Prefill: processing the input. Scales roughly linearly with input length. A 5K-token prompt prefills in 200-600ms on a hosted endpoint.
• Decode: generating output, one token at a time. This is the long tail. A 500-token output at 50ms per token is 25 seconds.

Two of these you can attack directly. Reduce prefill by reducing input length, mainly through prompt caching (covered in Part 7). Reduce decode by reducing output length (better prompting) or by reducing tokens-per-second cost (provider choice, smaller model, speculative decoding for self-hosting in Part 10).

For self-hosting: continuous batching

If you self-host with vLLM, SGLang, or TGI, you get continuous batching by default. Static batching, where requests wait for a batch to fill before processing, is dead. Continuous batching reshapes the batch at each generation step: completed sequences drop out, new ones join, the GPU stays busy.

This was the breakthrough that 10x'd throughput in 2023 and is now the industry default. Production benchmarks consistently show 2-3x improvement over static batching with no quality cost. If you self-host and your throughput is poor, check first that continuous batching is actually on.

The configuration knobs that matter most: --gpu-memory-utilization (raise to 0.95 from the default to give PagedAttention more KV cache room) and --max-num-seqs (the maximum concurrent sequences in flight). Tune those before anything else.

Prefill vs decode have different optimization profiles

This is the conceptual frame that ties everything together. An LLM call has two phases with very different characteristics.

Prefill is compute-bound and parallel. The GPU processes the entire input in one pass. Long prompts have heavy prefill. Batching helps because you can pack more input through.

Decode is memory-bandwidth-bound and sequential. Each new token requires a forward pass that depends on the previous token. Long outputs have heavy decode. Speculative decoding helps because it parallelizes the sequential bottleneck.

Understanding which phase dominates your workload tells you which optimization to reach for. Long prompts with short outputs: prefill-dominated, optimize input. Short prompts with long outputs: decode-dominated, optimize output generation. The frontier optimization, called disaggregated serving, runs prefill on one set of GPUs and decode on another. Worth knowing about, not worth implementing yourself unless you operate at very large scale.

The single rule that catches the most issues

p99 latency is usually dominated by queueing, not generation. If your p99 is bad, look at queue depth and concurrency limits before blaming the model. The fix is often raising your provider rate limit, adding retries on a separate provider, or capping concurrency on your side to keep queue depth low.

What to build this week

Add latency instrumentation to your harness:

• Capture TTFT, ITL, and end-to-end on every streaming call
• Report p50, p95, p99 for each metric, segmented by feature
• Add streaming to any user-facing endpoint that does not have it yet
• Load-test the endpoint with a tool like LLMPerf or Locust. Plot latency percentiles versus requests per second. Find your knee point, where latency starts climbing.
• Set SLOs on TTFT (under 1 second p95) and total latency (under 5 seconds p95 for chat use cases)

The load test almost always surprises teams. Most discover the knee is at much lower concurrency than they expected, usually because of provider rate limits, not model speed.

What is next

Part 7: prompt caching versus semantic caching. Two completely different mechanisms with similar-sounding names. Together they are the largest cost lever in your stack, and confusing them is how teams overspend.

Previous in the series

• Part 1: Harness Engineering
• Part 2: Context Engineering & Retrieval Quality
• Part 3: Structured Outputs and Fallback Chains
• Part 4: Evals
• Part 5: LLM Observability

References

1. vLLM documentation. The reference implementation for continuous batching and PagedAttention.
2. How continuous batching enables 23x throughput in LLM inference by Cade Daniel at Anyscale.
3. Baseten engineering blog. Consistently strong writing on production inference latency.
4. LLMPerf. The load testing tool that produces honest latency percentiles for LLM endpoints.
5. DistServe: Disaggregating Prefill and Decoding. The paper on the frontier optimization for large deployments.
6. Large Transformer Inference Optimization by Lilian Weng. Still the best technical overview of the underlying mechanics.