multi-router.aidocs

API Access

Rate Limit

The number of requests each API key can send per minute, and the response when a key exceeds it.

A rate limit caps how many requests an API key can send in a period of time. Multi-Router applies a rate limit to every key, which protects the organization's provider accounts and budgets from a runaway script or a leaked key. The limit counts requests over a rolling 60-second window, so a key that reaches it can send again as soon as earlier requests fall outside the window.

Limits

Each API key has the following limits:

EndpointCounts toward
POST /v1/chat/completionsInference, 300 requests per minute
POST /v1/messagesInference, 300 requests per minute
POST /v1/responsesInference, 300 requests per minute
Gemini Proxy endpoints under /v1beta/modelsInference, 300 requests per minute
Claude Proxy Bedrock endpoints, including token countingInference, 300 requests per minute
Claude Proxy Vertex endpoints, other than token countingInference, 300 requests per minute
POST /v1/messages/count_tokensToken counting, 300 requests per minute
Claude Proxy Vertex token countingToken counting, 300 requests per minute

Token counting has its own limit, which keeps inference capacity free for requests to models. Each key has one inference count and one token counting count, shared by every endpoint in the row.

A workload that signs in with Machine Identity has the same limits for each machine binding and subject. See Machine Identity.

The 429 response

A request over the limit is refused before it reaches a provider:

Response 429json
{
  "error": {
    "type": "rate_limit_error",
    "code": "rate_limit_exceeded",
    "message": "per-token rate limit exceeded"
  }
}

On Messages the response uses the Anthropic error format with the same type and message, and on a Claude Proxy's Bedrock and Vertex endpoints it uses their error format with the same message. The OpenAI and Anthropic SDKs retry 429 responses automatically with exponential backoff.

Other limits

Rate limits are one of several controls on usage, each with its own purpose and response:

ControlLimitsResponse
Rate limitRequests per key per minute.429 rate_limit_exceeded
BudgetSpending over a period, for the organization, chosen routers or router tags.429 budget_exhausted, with Retry-After
Provider rate limit on a static, fallback or weighted routerThe provider's own limits on the organization's provider key.Multi-Router retries, a fallback router moves to its next member, and otherwise the caller receives 429 upstream_rate_limit.
Provider rate limit on a Claude Proxy, Codex Proxy or Gemini ProxyThe provider's own limits on the key or login in use.The provider's 429, passed through as sent.

See Budgets & Spend Control.

AuthenticationInstalling the CLI
Was this helpful?