Model routing and fallback

What is model routing and fallback?

Model routing is deciding, per request, which model gets the job. A short billing question goes to a small, cheap model. A complaint with a ten-page attachment goes to a large one. A request carrying personal data stays on a deployment inside the EU. That decision is made before the call, on purpose, by a rule or a small classifier sitting in front of your models.

Fallback is what happens when the chosen model does not deliver. The provider returns an error, the call times out, you hit your rate limit, the prompt is longer than the model's context window, or the model has been retired since you wrote the code. A fallback sends the same request somewhere else: the same model in another region, an older version, a different provider, a cached answer, or a message saying the assistant is unavailable and a colleague will follow up.

Routing is the triage nurse at reception. Fallback is the backup generator.

The forms of routing

1. Static per endpoint. Your invoice extractor always calls the small model, your contract reviewer always the large one. No runtime decision, just a model ID in configuration per task. Most companies should start here and many never need more.

2. Rule-based on the input. The router looks at something it can measure without calling a model: prompt length, language, the customer's plan, a tag the calling application sets. Portkey's conditional routing works this way, with rules on request metadata and parameters and a required default when nothing matches. LiteLLM does the same with tags, so a request tagged free lands on one deployment and one tagged paid on another.

3. Learned difficulty scoring. A small classifier estimates how hard the question is and sends only the hard ones up. LMSYS, the group behind Chatbot Arena, showed in 2024 that a router trained on people's preferences between two models kept about 95 percent of the strong model's quality on a chat benchmark at up to three quarters less cost, and that the same router still worked when they swapped in a different pair of models underneath it.

4. Cascade. The small model answers first, and a scoring step decides whether that answer is good enough to keep or has to move up. Written up in 2023 as an LLM cascade, with large reported savings because most questions never reach the expensive model. The catch is the scorer: a cascade is only as good as its ability to tell a confident wrong answer from a right one.

5. Advisor. The newest form turns the cascade inside out. A small model runs the whole task and, when it gets stuck, calls a strong model as a tool, gets a plan back and continues. Anthropic shipped this in April 2026 as the advisor tool: the executor (Haiku or Sonnet) does the work, the advisor (Opus or Fable) reads the full transcript and returns a few hundred tokens of guidance at its own rate. In Anthropic's measurement, Haiku with an Opus advisor went from roughly 20 to 41 percent on a web research task, and max_uses caps how often the advisor may be called per request.

The forms of fallback

A fallback is triggered by a failure, so start from the failures you expect: a 429 past your rate limit, a 5xx on the provider's side, a context-length error, a refusal from a content filter. A retired model simply fails. Anthropic states that requests to retired models will fail, gives at least 60 days of notice, and took Claude Sonnet 4 and Opus 4 out of service on 15 June 2026. OpenAI promises at least six months for generally available models and as little as two weeks for previews.

The answer to each failure is one of five things.

• The same model elsewhere. Another deployment of the same model, in another region or on reserved capacity. This is the fallback with no change in output, and Microsoft's architecture guidance is firm about it: when you fail over or load balance, use the same model at the same version, and do not fail over from version X to X+1, because the behaviour change surprises the clients.

• An older or smaller version. Cheaper to keep warm, but now the answers change and nobody sees it unless you measure.

• Another provider. LiteLLM, Portkey and OpenRouter all let you name a second provider's model as the next target. OpenRouter bills the model that actually answered and returns its name in the model field of the response, so you can at least see when it happened.

• A cached answer. For questions that repeat, a semantic cache can serve the last good answer while the model is down. Only for content that is not personalised.

• Graceful degradation. A fixed message, a link to the FAQ, a ticket for a colleague. Every system needs this one, because the four above can all fail at the same time.

Two controls sit around this in any gateway. A cooldown or circuit breaker pulls a failing backend out of rotation for a while: LiteLLM cools a deployment down after three failures by default, and Azure API Management lets you set a failure count over an interval plus a trip duration. And the breaker should respect the Retry-After header, because as Microsoft warns, a 429 from Azure OpenAI can ask you to wait a full day.

Routing versus fallback: when is the decision made?

Every product page bundles the two, and they differ on one dimension that shapes your design.

Routing decides before the call, on purpose, from the request itself. You can test that decision offline against a pile of past requests and know exactly which model each one would get. Its failure mode is a bad estimate: the small model got a hard question.

Fallback decides after a failure, in reaction to the provider. You cannot test it on past requests, only by making the failure happen. Its failure mode is a silent switch: the answer came from a model you did not plan for, and nobody noticed.

So routing needs an eval per route and fallback needs an alert per trigger. If your monitoring cannot tell you how many requests went to the fallback yesterday, you do not have a fallback strategy, you have a surprise.

Where the logic lives

In your own code, which is fine for one application and one model. In a router library you run yourself, such as the LiteLLM proxy or a self-hosted Portkey, where routing strategies (shuffle, lowest latency, lowest cost, least busy), fallback lists per model and cooldowns sit in one configuration file. Or in a managed gateway such as the AI gateway in Azure API Management, whose backend pools balance round robin, by weight or by priority, with a lower-priority group receiving traffic only once every backend above it has tripped. Hosted routers such as OpenRouter do the same as a service, spreading requests by price and recent uptime unless you sort by throughput or latency. Once a second application starts calling models, this belongs in a gateway rather than in each application separately.

A worked example: the support assistant

A company answers 20,000 customer mails a month with an assistant that drafts replies for the support team.

The rule. A mail under 150 words that a classifier labels billing or order status goes to the small tier. Anything labelled a complaint, anything with an attachment, and anything a sentiment check flags as angry goes straight to the large tier. Say that puts 70 percent of the mails on the small tier.

The escalation. The small tier returns a confidence score in its structured output, and below a threshold the mail is re-sent to the large tier. The eval tells you whether the threshold is right: sample the mails that were not escalated and count how many the large tier would have answered differently.

The fallback. Both tiers run on two deployments in the EU data zone. A 429 or a timeout moves the call to the second one, and three failures in a minute take a deployment out of rotation. If both tiers fail on both deployments, the customer gets a fixed reply that the mail arrived and a colleague will answer, and the mail lands in a human queue tagged as assistant unavailable.

Four numbers belong on the dashboard: share of mails per route, escalation rate, fallbacks per day, and the eval score per route.

What to watch out for with model routing and fallback

The small model does not know it is wrong. A confidence score from the model itself is a hint, not a measurement. The only reliable answer is an eval per route on your own documents, rerun whenever anything changes.

Silent quality drops after a fallback. An older version or another provider gives different answers. Log which model answered which request, and decide up front whether the user is told. For a draft a colleague reviews, probably not. For an answer that goes straight to a customer, a short note that the assistant is running in reduced mode is honest and cheap.

Data residency of the fallback. A fallback to a global deployment or a US provider moves personal data across a border you promised to stay inside. On Azure, Global Standard deployments may process prompts in any Azure region, and only Data Zone or regional deployments keep processing within the EU.

Routing rules that outlive the model generation. A rule written for last year's small tier still sends work upstairs that this year's small tier handles fine. The whole ladder of model tiers shifts with each generation, so rerun the routing eval when a new one lands.

Cost caps. A cascade that escalates too often, or an advisor called at every step, quietly pays for two models per request. Put a token budget per application on the gateway and cap the advisor calls.

The second provider's terms. Your fallback provider has its own retention periods, its own clauses about training on your data and its own subprocessors. OpenRouter lets you exclude providers that store data and route only to zero-data-retention endpoints. On a direct contract you read the terms yourself, before the outage.