Model Routing and Graceful Fallback: Picking the Right Model for Each Request

Routing is the cheapest optimization you are not doing

Most teams pick a model and use it for everything. They feel sophisticated when they upgrade from one model to a slightly larger one across the board. They are leaving 40-60% of their bill on the table, and they are paying premium-model latency for traffic that did not need it.

The reason is that not every request is equally hard. A user asking "what's my account balance" does not need the same model that handles "draft a contract addendum reflecting these three clauses." Routing decides which model gets which request, and a well-designed routing layer is the single highest-ROI optimization most teams have available, behind only evals.

This is Part 11 of the AI Engineer Series. We will cover the three main routing patterns (cascading, classifier-based, learned), the fallback patterns that keep you alive when a provider goes down, and the failure modes that make routing harder than it looks.

The cascade pattern

Cascading is the simplest and often the best. Run the request through a cheap model first. If the result passes a confidence check, return it. If it fails the check, escalate to a more expensive model. Repeat as needed.

The confidence check is the entire trick. Without one, you have no idea whether the cheap model's answer was acceptable. Possible signals:

• Schema validity (from Part 3). If the cheap model failed to produce a valid structured output, escalate.
• An explicit self-rating field in the response. The cheap model says "confidence: low" and triggers escalation.
• A separate verifier model that grades the cheap model's output on a rubric.
• Token-level log probabilities, if your provider exposes them. Low-probability tokens correlate with model uncertainty.

A cascade with a 70% pass rate on the cheap model saves you roughly 70% of cost on those 70% of requests. The math compounds. If the cheap model is 1/20th the cost and 70% of traffic stops there, your blended cost is something like 0.7 * 0.05 + 0.3 * 1.0 = 0.335, a roughly 3x reduction. With careful tuning, cascades regularly deliver 50-70% cost savings on traffic that was previously all-premium.

The risk is the latency on the escalated 30%. They pay the cheap model's latency plus the expensive model's latency. Always measure end-to-end p95 with and without the cascade. If escalations make p95 worse and tail latency matters more than cost on this surface (it does for interactive features, doesn't for batch), the cascade is not the right tool here.

Classifier-based routing

Cascading is reactive: try cheap, escalate on failure. Classifier-based routing is proactive: classify the request upfront and dispatch to the right model directly. This avoids the latency penalty of cascading on hard requests, at the cost of an extra classification call.

The classifier can be:

• A small fine-tuned model (BERT-class, or a 1-3B LLM) trained on labeled examples of "this query needs the cheap model" vs "this needs the expensive model"
• An LLM call with a tight prompt that returns just the route choice
• Heuristic rules over surface features (token count, presence of code, language, etc.)

The classifier itself has to be cheap enough that it doesn't eat the savings. A 100ms classification call that saves a 5-second expensive-model call is fine; a 2-second classification call that saves a 5-second call is not.

The hardest part is labeling the training data for the classifier. You need examples of requests with ground-truth "right model" labels, which usually means running both models on a sample and grading the outputs. This is where Part 4 evals pay for themselves twice: the same eval set that grades model output quality also tells you which model was sufficient for each input.

RouteLLM and learned routers

RouteLLM, from LMSYS, is the most public example of a trained routing layer. It learns from pairwise preference data (which model wins on which kind of query) to predict the cheapest model that will hit a quality threshold. The original paper reports 85% cost savings on MT-Bench-style traffic with no measurable quality drop, by routing a substantial fraction of requests to Mixtral 8x7B instead of GPT-4.

The general technique generalizes. If you have observability data (Part 5) and judge scores (Part 4) on a meaningful sample of past traffic, you can train a router. The features that work best are:

• Embeddings of the input (BGE, E5, or whatever your retrieval stack already uses)
• Lightweight surface features: input length, output-length estimate, presence of code/tables, language detection
• User or feature tags (some features have predictably harder traffic than others)

Logistic regression or a gradient-boosted tree on these features gets you most of the way. You do not need a neural router. You do need a steady stream of labeled feedback to keep the router calibrated as traffic and models shift.

Provider failover, the other half of routing

Routing for cost is one axis. Routing for availability is the other. Every major provider has had multi-hour outages this year. If your service goes down when theirs does, you have a single point of failure you can fix in a weekend.

The pattern is straightforward: define a primary provider and one or two fallbacks. On a request, attempt the primary. On a timeout or 5xx, retry the fallback. Cap total retries so a misbehaving provider doesn't blow your latency budget.

async def call_with_failover(prompt, providers, timeout_s=10):
    for provider in providers:
        try:
            return await asyncio.wait_for(
                provider.call(prompt),
                timeout=timeout_s
            )
        except (TimeoutError, ProviderError) as e:
            log_failover(provider.name, str(e))
            continue
    raise AllProvidersFailedError()

The gotchas are the parts the snippet hides. Models behave differently across providers, even when they nominally have the "same" name. Anthropic on Bedrock vs Anthropic direct has different rate limits, slightly different latency profiles, and occasionally different model versions live at the same name. Your fallback model needs its own eval pass to confirm it produces acceptable output for your tasks, or you will discover during an outage that the fallback regresses quality in ways your users notice.

Litellm, OpenRouter, and Portkey are the main multi-provider abstractions. They handle the API differences and let you configure failover declaratively. For most teams, picking one of these is faster than building your own.

Graceful fallback when nothing works

What if all your providers fail? Your harness (Part 1) should have a degraded-mode response ready. The options, in order of preference:

• Serve a cached response if the request matches a cache key (Part 7)
• Serve a "we're having trouble, try again in a moment" message that is honest, friendly, and doesn't pretend to be an AI response
• Fall back to a deterministic, non-LLM path if one exists (e.g., for search-style features, return raw search results)
• Queue the request for retry and return an async-status response

The wrong answer is to silently retry forever or to return an obviously broken response. Both erode user trust in ways that take much longer to recover from than a clean "this is temporarily unavailable" message. The right answer depends on the surface, but every LLM feature should have a documented degraded mode that has been tested at least once. The test is to disable all providers in staging and confirm that the user-facing behavior is acceptable.

The metrics that tell you routing is working

Routing introduces complexity. The case for keeping it is data:

• Blended cost per request, broken down by route choice. This is the headline number. If the cheap-model route is taking 70% of traffic at acceptable quality, you are winning.
• Escalation rate, for cascades. If it climbs from 30% to 50% over a week, either traffic has shifted harder or the cheap model has regressed. Either way, you need to know.
• Quality parity, measured via online judge (Part 5) on routed vs unrouted traffic. The whole bet is that routing is free quality-wise. Verify.
• Failover rate, per provider. Sustained high failover is a contract conversation with the provider, not a quiet engineering fix.
• Tail latency, p95 and p99. Cascades push tails up; classifiers push tails down. Track both.

These metrics live in the same observability stack you built in Part 5, with the same per-feature tagging you set up in Part 8. None of this requires new infrastructure if you have done the earlier work.

The order to build this

Most teams should build routing in this order:

1. Provider failover first. This is purely defensive, low complexity, and prevents you from going down when a provider does. No clever logic, just a try/catch chain.
2. Cascade second. Simple enough to ship in a week, savings are immediate, and the engineering team learns how to monitor blended cost without anything stochastic in the loop.
3. Classifier-based routing third, once you have eval-graded traffic data to train on. This is where the cost wins compound.
4. Learned routers fourth, if and only if scale justifies the model-training overhead. Most teams never need this.

The reverse order, starting with a learned router, is where teams burn a quarter building a model and a feedback loop before they ever shipped the simple cascade that would have captured 80% of the savings.

What is next

Part 12 is about agents. Specifically, about how to keep them from running forever, burning your budget, and doing things you did not ask for. We will cover the five budgets every agent run needs, tool-use safety, prompt injection defenses, and what the OWASP LLM Top 10 has to say about it.

Previous in the series

• Part 1: Harness Engineering
• Part 2: Context Engineering & Retrieval Quality
• Part 3: Structured Outputs and Fallback Chains
• Part 4: Evals
• Part 5: LLM Observability
• Part 6: Latency and Throughput Engineering
• Part 7: Prompt Caching vs Semantic Caching
• Part 8: Cost Attribution Per Feature
• Part 9: The KV Cache
• Part 10: Speculative Decoding and Quantization

References

1. RouteLLM: Learning to Route LLMs with Preference Data by Ong et al. The LMSYS paper that productionized learned routing.
2. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance by Chen et al. The original cascade paper, with the cost/quality tradeoffs worked out.
3. LiteLLM routing documentation. The most-used multi-provider abstraction; covers failover, retries, and load balancing.
4. OpenRouter provider routing. Declarative provider preferences with built-in failover.
5. Portkey fallbacks documentation. Another mature gateway with first-class fallback semantics.
6. LLM Patterns by Eugene Yan. Section on cascading and routing covers the patterns at a higher level than any one paper.