multi-router.aidocs

Budgets & Spend Control

OverviewEnterprise

Capping and tracking spend on AI models, with notifications and blocks at the points you choose.

A budget tracks what an organization spends on AI models over a period and acts at the points you set. Each budget covers part of the organization's traffic, such as everything, a list of routers, or every router with a tag, and holds caps on spend, in total, per router and per model. Rules on the budget decide what happens as spend approaches a cap: an email, a webhook, or blocking requests until the period resets.

Parts of a budget

A budget is made of these parts:

PartPurpose
ScopeWhich traffic the budget covers: the whole organization, a list of routers, or routers with a tag.
PeriodDaily, weekly or monthly, reset at 00:00 UTC, or lifetime, which never resets.
CapsA total cap, caps per router, and caps per model.
RulesWhat happens at a point of a cap: notify by email or webhook, or block.

Tracking and enforcing

A budget's Notify rules track spend and send notifications; a Block rule refuses requests once spend reaches its line, until the period resets or the cap is raised. Multi-Router checks budgets before each request and updates spend every few seconds, so a block takes effect within seconds of the line. Requests already running when the line is crossed finish and are counted, so spend can pass a cap by the cost of those requests.

What counts

Spend is the cost of each inference request, worked out from the model's price as in Traffic. Every request a router serves counts: failed requests count $0, and a stream the caller closed early counts the usage that had arrived. Token counting is free and stays outside budgets. Requests to models with no price count at $0. See Cost.

Several budgets

Every enabled budget that covers a request is checked on its own, and a request is refused when any one of them blocks it. A router in several budgets therefore stops at the lowest line among them. Raising one budget lifts only that budget's block.

Budgets from tools policies

Each tool in a tools policy has a budget, created and managed by the policy, that caps each person's spend on the tool. These budgets appear on the Budgets page as Managed by policy, and their caps and rules change by editing the policy. See Tools Policies.

The other pages in Budgets & Spend Control cover each part in detail:

PageCovers
Creating a BudgetScopes, caps and periods.
Rules and AlertsNotifications and blocks.
WebhooksSending budget alerts to your own systems.
Overrides and RequestsRaising a cap for one period, and requests for more.
Status and UsageThe Budgets page, status labels and usage charts.
When a Budget BlocksWhat callers receive, on each endpoint.
Budgets APIManaging budgets with /v1/budgets.
TroubleshootingCreating a Budget
Was this helpful?