multi-router.aidocs

Providers & Models

Supported Providers

The model providers Multi-Router routes to, the settings each provider key needs, the request features each provider supports, and how requests and responses are translated.

Multi-Router routes requests to several kinds of model provider. Model publishers such as Anthropic, OpenAI and Google serve their own models. Inference platforms such as Fireworks, Groq and Together serve open models. Cloud platforms such as AWS Bedrock and Google Vertex serve models from many publishers across regions. Multi-Router also routes to custom endpoints that implement the OpenAI Chat Completions API. Your application calls every provider through the same two endpoints, Chat Completions and Messages, and Multi-Router translates each request into the provider's own API and each response back.

Providers

A provider is a company or cloud platform that serves models to Multi-Router, identified by a provider id that router configurations use and the x-mr-provider response header returns. A router whose provider does not serve its model returns provider_not_offered.

ProviderProvider idPublishersProvider key settings
AnthropicanthropicAnthropicAPI key
OpenAIopenaiOpenAIAPI key and Region (data residency, default Global)
Google GeminigeminiGoogleAPI key
CoherecohereCohereAPI key
AWS BedrockbedrockAmazon, Anthropic, DeepSeek, Meta, Mistral, Moonshot, Nvidia, OpenAI, Qwen, Z.aiAWS Region, and an access key (SigV4) or Bedrock API key. See AWS Bedrock.
Google VertexvertexAnthropicService account, project and location. See Google Vertex.
DeepSeekdeepseekDeepSeekAPI key
FireworksfireworksDeepSeek, Moonshot, OpenAI, Z.aiAPI key
GroqgroqMoonshot, OpenAIAPI key
MistralmistralMistralAPI key
TogethertogetherDeepSeek, Moonshot, OpenAI, Z.aiAPI key
xAIxaixAIAPI key
Z.aizaiZ.aiAPI key
Custom endpointcustomAny model the endpoint servesBase URL, API format and API key. See Custom endpoints below.

Provider endpoints

The provider endpoints are the hosts Multi-Router calls for each provider, with the authentication it uses and the regions it serves. A self-hosted installation allows outbound HTTPS on port 443 to the hosts of the providers it uses.

ProviderUpstream endpointAuthenticationRegions
Anthropichttps://api.anthropic.com/v1/messagesx-api-keyglobal, us
OpenAIhttps://api.openai.com/v1, and https://us.api.openai.com/v1, https://eu.api.openai.com/v1 or https://ae.api.openai.com/v1 for data residencyBearer tokenglobal, us, eu, ae
Google Geminihttps://generativelanguage.googleapis.com/v1betax-goog-api-keyglobal
Coherehttps://api.cohere.com/v2/chatBearer tokenglobal
AWS Bedrockhttps://bedrock-runtime.{region}.amazonaws.comAWS SigV4 or Bedrock API keyAWS regions
Google Vertexaiplatform.googleapis.com, aiplatform.us.rep.googleapis.com, aiplatform.eu.rep.googleapis.com and {region}-aiplatform.googleapis.com, plus oauth2.googleapis.comAccess token from the service accountGoogle Cloud locations
DeepSeekhttps://api.deepseek.com/v1Bearer tokenglobal
Fireworkshttps://api.fireworks.ai/inference/v1Bearer tokenglobal
Groqhttps://api.groq.com/openai/v1Bearer tokenglobal
Mistralhttps://api.mistral.ai/v1Bearer tokenglobal
Togetherhttps://api.together.xyz/v1Bearer tokenglobal
xAIhttps://api.x.ai/v1Bearer tokenus-east-1, us-west-2, eu-west-1
Z.aihttps://open.bigmodel.cn/api/paas/v4Bearer tokenglobal
Custom endpoint{base URL}/chat/completionsBearer tokenSet by the custom model

DeepSeek, Fireworks, Groq, Mistral, Together, xAI, Z.ai and custom endpoints use the OpenAI Chat Completions format, and Multi-Router sends them the same request it sends OpenAI, with max_tokens sent as max_completion_tokens. Regions are described in Regions & Variants.

Provider-specific behavior

ProviderBehavior
AnthropicSystem and developer messages anywhere in the conversation become the system prompt, and tool results are sent as user messages. Claude requests on Google Vertex and AWS Bedrock follow the same behavior.
Google GeminiSystem and developer messages are combined into Gemini's system instruction. Gemini returns no tool call id, so Multi-Router uses the function name as the id, and a tool result is matched to its call by function name.
CohereCohere receives the text of each message. A json_schema response format is sent as Cohere's JSON mode with the schema, and top_p and top_k are sent as p and k.
AWS BedrockImage URLs are downloaded and sent to Bedrock as image data. See AWS Bedrock.
Google VertexServes Anthropic models only, and requests for other publishers' models return publisher_not_supported. See Google Vertex.
xAIThe region is a label for the request: every region of a model has the same price and is served from the same endpoint.

Custom endpoints

The custom provider sends requests to an endpoint that implements the OpenAI Chat Completions API, such as a self-hosted model server. Its provider key holds the Base URL, the API format (OpenAI Chat Completions) and the API key, and Multi-Router appends /chat/completions to the base URL, which must use HTTPS and a public DNS name, with no port, credentials, query string or fragment. A router on a custom endpoint can name any model the endpoint serves, and requests skip the catalog checks. Requests carry no price unless the model is registered with a price in Custom Models & Pricing.

Feature support

Feature support shows which request features each provider supports: streaming responses, tool calls, parallel tool calls, image input, structured output that follows a JSON schema, reasoning, and prompt caching. Per model means some of the provider's models support the feature, and the Models page lists the features of each model. A request that uses a feature its model lacks returns an error named for the feature; see Model Catalog.

ProviderStreamingToolsParallel tool callsImagesStructured outputReasoningPrompt caching
AnthropicYesYesYesYesClaude 4.5 and laterClaude 3.7 and laterYes
Google VertexYesYesYesBase64 onlyClaude 4.5 and laterClaude 3.7 and laterYes
AWS BedrockPer modelPer modelClaude modelsPer modelNoClaude 3.7 and laterClaude models
OpenAIPer modelPer modelPer modelPer modelPer modelPer modelPer model
Google GeminiPer modelYesNoYesPer modelPer modelPer model
CohereYesYesNoPer modelYesPer modelNo
DeepSeekYesYesNoNoYesYesYes
FireworksYesYesNoPer modelYesPer modelPer model
GroqYesYesNoNoYesPer modelNo
MistralYesYesNoPer modelYesNoNo
TogetherYesYesNoPer modelYesPer modelPer model
xAIYesYesYesYesYesPer modelYes
Z.aiYesPer modelNoPer modelPer modelPer modelPer model

Claude features by endpoint

Claude models on Anthropic, Vertex and Bedrock support some features on only one of the two endpoints, as follows:

Feature/v1/chat/completions/v1/messages
Structured outputresponse_format with json_schema, on Anthropic and VertexNot available
ReasoningNot availablethinking with type: "enabled" and budget_tokens, returned as thinking blocks
Prompt cachingNot availablecache_control on content blocks, system blocks and tools
Maximum output tokensmax_tokens or max_completion_tokens; 1,024 when neither the request nor the router sets onemax_tokens is required

Input content

Input content is the set of content types a request message can contain. Requests can contain text, images, tool calls and tool results; images are sent as a URL or, on Chat Completions, as a data: URL with base64 data. Google Vertex accepts base64 images only, AWS Bedrock receives URL images after Multi-Router downloads them, and Cohere receives text only. On Chat Completions, a tool message keeps its text parts, and a content part of any type other than text and image returns not_supported. The Messages endpoint also accepts thinking blocks, and removes other block types, such as redacted_thinking and search result blocks, before sending the request.

Translation between API formats

Translation is the conversion of a request into the provider's API and of the response back into the format of the request. Multi-Router accepts OpenAI Chat Completions requests at /v1/chat/completions and Anthropic Messages requests at /v1/messages, and either format can target any provider. Messages, system prompts, text, tools, tool choice, tool calls and results, stop sequences, temperature, top_p and maximum tokens are converted for every provider. Other fields are converted when the provider's API has an equivalent and removed otherwise, and fields Multi-Router does not recognize are removed.

EndpointProviderFields removed
Chat CompletionsOpenAIImage detail. Every other recognized field is sent.
Chat CompletionsOpenAI-compatible providersImage detail. Every other recognized field is sent.
Chat CompletionsClaude modelsreasoning_effort, seed, logprobs, top_logprobs, penalties, logit_bias, parallel_tool_calls, prompt_cache_key, prompt_cache_retention, service_tier, store, metadata, user, safety_identifier, verbosity, prediction, tool strict, image detail
Chat CompletionsGoogle Geminireasoning_effort, prompt cache fields, parallel_tool_calls, penalties, logprobs, top_logprobs, logit_bias, service_tier, store, metadata, user, safety_identifier, verbosity, prediction, tool strict, non-text tool results
Chat CompletionsCohereImages, reasoning_effort, prompt cache fields, parallel_tool_calls, logprobs, top_logprobs, logit_bias, service_tier, store, metadata, user, safety_identifier, verbosity, prediction, tool strict
Chat CompletionsNon-Claude models on BedrockEvery field outside the converted set.
MessagesEvery provideroutput_config, service_tier, container, mcp_servers, context_management, disable_parallel_tool_use
MessagesOpenAItop_k, thinking and thinking blocks, cache_control, metadata, non-text tool result content. A tool result with is_error is sent with a [Tool Error] prefix.
MessagesOpenAI-compatible providerstop_k, thinking and thinking blocks, cache_control, metadata, non-text tool result content. A tool result with is_error is sent with a [Tool Error] prefix.
MessagesGoogle Geminithinking and thinking blocks, cache_control, non-text tool result content
MessagesCoherethinking and thinking blocks, cache_control, non-text tool result content, images

The anthropic-beta request header is forwarded to Anthropic and Google Vertex. Reasoning text from the model is omitted from Chat Completions responses. Tool call ids pass through unchanged in both directions, so a tool result names the id the provider issued: toolu_ ids from Claude, call_ ids from OpenAI, and the function name from Gemini.

The following requests return an error:

RequestError
Chat Completions with n greater than 1not_supported
Chat Completions with a name on a messagenot_supported
Chat Completions with a tool other than a functionnot_supported
Chat Completions with a content part other than text and imagenot_supported
A json_object or json_schema response format sent to a Claude model before 4.5structured_output_not_supported
A json_object response format sent to Claude 4.5 or laterinvalid_request
Any response_format sent to Claude on AWS Bedrockinvalid_request
Messages with thinking enabled and no budget_tokensinvalid_request_error
Messages with a system role inside messagesinvalid_request_error

Streaming

Streaming returns the response as server-sent events in the format of the endpoint, whatever format the provider streams in. Multi-Router always asks the provider for token usage, so Traffic records token counts whether or not the request asks for them.

Chat CompletionsMessages
Eventsdata: {chat.completion.chunk} lines, ending with data: [DONE]message_start, content_block_start, content_block_delta, content_block_stop, message_delta, message_stop, and ping
KeepaliveA : keepalive comment after 15 seconds without dataThe same
UsageA final chunk with empty choices and usage, when the request sets stream_options.include_usagemessage_delta.usage
Tool callsThe first chunk of each call carries id, name and index, and later chunks carry argument fragmentscontent_block_start with tool_use, then input_json_delta
ReasoningOmittedthinking_delta and signature_delta
Errors after the stream startsdata: {"error": ...}, then data: [DONE]event: error

Token usage

Token usage is the count of tokens the provider reports, returned in the format of the endpoint:

Chat CompletionsMessages
prompt_tokensinput_tokens
Input tokens as the provider reports them. Claude providers exclude cache reads and writes, and OpenAI, OpenAI-compatible providers and Gemini include cached tokens.
completion_tokensoutput_tokens
Output tokens, including reasoning tokens.
total_tokensNot returned
Input and output tokens combined.
prompt_tokens_details.cached_tokenscache_read_input_tokens
Tokens read from the prompt cache.
Not returnedcache_creation_input_tokens
Tokens written to the prompt cache.
Not returnedcache_creation
Tokens written to the prompt cache, broken down by cache lifetime.
completion_tokens_details.reasoning_tokensNot returned
Reasoning tokens, when the provider reports them.
Not returnedinference_geo
The region Anthropic reports the request ran in.

Response fields

Response fields follow the format of the endpoint, with the following values set by Multi-Router. The model field returns the value the request sent, such as the router name, and the x-mr-upstream-model and x-mr-upstream-wire-id headers name the model that served it. The id is the provider's response id, prefixed with chatcmpl- on Chat Completions. created is the time Multi-Router built the response, and Messages responses return stop_sequence as null. Stop reasons are mapped as follows:

Provider reasonChat Completions finish_reasonMessages stop_reason
end_turnstop, or tool_calls when the response has tool callsend_turn
stopstop, or tool_calls when the response has tool callsend_turn
completestop, or tool_calls when the response has tool callsend_turn
stop_sequencestopstop_sequence
max_tokenslengthmax_tokens
lengthlengthmax_tokens
tool_usetool_callstool_use
tool_callstool_callstool_use
content_filtercontent_filterend_turn
safetycontent_filterend_turn
Any other reason, such as refusalstopend_turn

Provider errors and retries

A provider error is a failure the provider returns after Multi-Router sends the request. Multi-Router returns it with a Multi-Router error code and an upstream object holding the provider, the provider's error code and its HTTP status; the provider's own message is not passed through, because it can contain account details.

Provider error on Chat Completionsjson
{
  "error": {
    "message": "upstream provider rate limited this request after multiple retries",
    "type": "upstream_error",
    "code": "upstream_rate_limit",
    "upstream": { "provider": "anthropic", "code": "max_attempts_exceeded", "status": 529 }
  }
}
ErrorStatusProvider responses
upstream_invalid_request400The provider rejected the request as invalid.
upstream_context_length400The prompt exceeds the model's context window.
upstream_content_filter400The provider's content policy blocked the request.
upstream_model_not_found400The provider does not offer the model to the account.
upstream_invalid_credentials502The provider rejected the provider key.
upstream_insufficient_quota402The provider account has no remaining credit or quota.
upstream_rate_limit429The provider rate limited the request or timed out, and retries ran out.
upstream_server_error502The provider returned a server error or was overloaded (5xx or 529), and retries ran out.
upstream_transport503The connection failed while the response was being read.
upstream_otherThe provider's statusAny other provider error.

On the Messages endpoint, the same failures use Anthropic error types chosen by status: invalid_request_error for 400, rate_limit_error for 429, overloaded_error for 503 and api_error otherwise, with the upstream object included.

Multi-Router retries a provider request that fails with HTTP 429, 500, 502, 503, 504 or 529, a timeout or a connection failure, for up to five attempts. Waits double from one second with up to 25% random jitter, up to 60 seconds, and a whole-second Retry-After from the provider replaces the computed wait. Retries stop when another attempt would run past five minutes from the first, and a 429 that reports exhausted credit is returned at once. These rules apply to every provider and to custom endpoints, and Claude Proxy requests are sent once, without retries.

Limits

Limits are the size and time bounds Multi-Router applies to provider requests:

LimitValueWhen exceeded
Chat Completions request body4 MiB413 body_too_large
Messages request body32 MB413 request_too_large
Time to the provider's response headers, per attempt120 secondsThe attempt is retried. Long generations use streaming.
Total time across retries5 minutesRetries stop.

Claude Code and Claude Desktop

A Claude Proxy forwards requests from Claude Code and Claude Desktop to Anthropic, AWS Bedrock or Google Vertex in their native API formats, replacing only the model id, so every feature of those APIs is available. See Agents & Editors.

OverviewProvider Key
Was this helpful?