Routing
Weighted Router
Split traffic across static routers by weight, for A/B tests and gradual rollouts.
A weighted router splits traffic across its member static routers. On each request it picks a member at random in proportion to the members' weights, so members weighted 90 and 10 receive about 90% and 10% of requests. Changing the weights changes the split at once, without changing clients, which suits A/B tests between models or providers and gradual rollouts of a new model. When the picked member fails, the request moves to another member, as in a fallback router.
Members and weights
A member is a static router in the same organization with a weight, a whole number of 1 or more. The weights do not need to add up to any total: each member's share is its weight divided by the sum of all weights. A weighted router has between 1 and 50 members, each listed once. A member added in the dashboard starts with weight 1, and the router's Members panel shows each member's share as a percentage of the total.
How a member is selected
Multi-Router draws a new random order of the members for every request, weighted by their weights, and tries the members in that order. The first member is picked with probability equal to its share, and when it fails, the request moves to the next member in the same order. Requests are independent, so a caller is not kept on one member across requests. The failures that move a request to the next member are the same as for a fallback router; see Fallback Router.
Headers, settings and Traffic
A weighted router holds its name, description, tags and members, and each request takes its model, provider key, region and settings from the member that serves it. Responses carry x-mr-policy: weighted (or header when selected with x-mr-router), x-mr-composer with the weighted router's id, and x-mr-attempts. In Traffic, the Router column shows the member that served each request, and the Type column shows weighted. When a weighted router is selected with x-mr-router, the request's model is ignored and each member uses its own model.
Creating a weighted router
A weighted router is created on the Routers page:
- Select New router, choose Weighted Router under Routers, and select Continue.
- Under Configure, enter a Name, an optional Description and Tags, add members with Add Static router, and set each member's weight.
- Select Review, check the members and weights, and select Create router.
To move more traffic to the new model, change each member's weight with PATCH /v1/routers/{id}/members/{key}, where the key is the member's router id, or change the weights in the router's Members panel.
Errors
Creating and changing a weighted router returns the following errors, each with status 400:
| Error | Cause |
|---|---|
invalid_weight | A member has no weight, or a weight of 0. |
invalid_json | A weight is not a whole number. |
missing_field | The members list is missing or empty. |
too_many_members | The router has more than 50 members. |
invalid_member | A member is not a static router in the organization. |
duplicate_member | A member is listed twice. |
member_exists | A member add names a router that is already a member. |
field_not_supported_for_kind | The body sets a field that does not apply, such as prompt. |