Routing
Fallback Router
Try a list of static routers in order until one succeeds, for failover across models and providers.
A fallback router holds an ordered list of member static routers and tries them in order until one returns a response. When the first member's provider is rate limited, unavailable or misconfigured, the request moves to the next member, so callers are not affected when a member fails. Each member brings its own model, provider, provider key, region and settings, so a fallback router can fail over across models, providers and regions, such as from Claude on Anthropic to the same model on AWS Bedrock.
Members
A member is a static router in the same organization. A fallback router has between 1 and 50 members, each listed once; members are static routers only. Members are tried in the order they are listed, top first, and a disabled or deleted member is skipped. A member keeps serving its own callers when it is also part of a fallback router, and a static router can be deleted only after it is removed from every fallback and weighted router.
When a request moves to the next member
Each failure either moves the request to the next member or returns the error to the caller:
| Failure | Result |
|---|---|
| The member router is missing or disabled. | Moves to the next member. |
| A budget refuses the member's model. | Moves to the next member. |
| The provider key is missing, disabled or for another provider. | Moves to the next member. |
The bound prompt cannot be applied, such as missing_prompt_variables. | Moves to the next member. |
A catalog check fails, such as region_not_served or tools_not_supported. | Moves to the next member. |
| The provider times out or the connection fails. | Moves to the next member. |
| The provider returns 429. | Moves to the next member. |
| The provider rejects the credentials (401). | Moves to the next member. |
| The provider does not offer the model (404). | Moves to the next member. |
| The provider returns a 5xx error after retries. | Moves to the next member. |
The request is malformed or uses an unsupported shape (invalid_request, not_supported). | Returns the error. |
| The provider rejects the request as invalid (400). | Returns the error. |
| The provider returns another 4xx error, such as 403, 409, 413 or 422. | Returns the error. |
| The client disconnects. | Ends the request. |
When every member fails, the response carries the last member's error, or 429 budget_exhausted when budgets refused every member. A streamed response commits to a member when the provider sends its first event; an empty stream, or a failure from the table above, before then moves to the next member, and an error after it arrives as an error event in the stream.
Settings
A fallback router holds its name, description, tags and members. The model, provider key, region, prompt binding, parameter defaults and request logging of a request come from the member that serves it, and the router's page shows Model, Prompt and Request Logging panels marked Configured on each member. When a fallback router is selected with x-mr-router, the request's model is ignored and each member uses its own model.
Headers and Traffic
Successful responses through a fallback router carry x-mr-policy: fallback (or header when selected with x-mr-router) and the provider, model and region headers of the member that served the request, along with the following:
| Header | Value |
|---|---|
x-mr-composer | The fallback router's id, also when the request named it through an alias. |
x-mr-attempts | The number of members tried. |
In Traffic, the Router column shows the member that served the request, the Type column shows fallback, and the Attempts column shows the members tried; the Type and Attempts columns are hidden by default and turned on with Columns. Budgets apply to the fallback router and the model of each member it tries.
Creating a fallback router
A fallback router is created on the Routers page:
- Select New router, choose Fallback Router under Routers, and select Continue.
- Under Configure, enter a Name, an optional Description and Tags, and add members with Add Static router. Order them with Move up and Move down.
- Select Review, check the numbered members, and select Create router.
On the router's page, the Members panel lists the members in order; Edit adds, removes and reorders them, and Save members applies the change. With the API, send POST /v1/routers with the member router ids in order in routing.routers:
Members change through /v1/routers/{id}/members: POST adds a member at an optional zero-based position, PATCH on /members/{key} moves it to a new position, and DELETE removes it. The key is the member's router id.
Errors
Creating and changing a fallback router returns the following errors, each with status 400:
| Error | Cause |
|---|---|
missing_field | The members list is missing or empty. |
too_many_members | The router has more than 50 members. |
invalid_member | A member is not a static router in the organization. |
duplicate_member | A member is listed twice. |
invalid_weight | A member of a fallback router has a weight. |
member_exists | A member add names a router that is already a member. |
field_not_supported_for_kind | The body sets a field that does not apply, such as prompt. |