multi-router.aidocs

Observability

Overview

What Multi-Router records about each request, and where to see usage, latency, errors and cost.

Observability in Multi-Router is the record of every request that passes through a router: who sent it, which router and model served it, how long it took, how many tokens it used, what it cost, and whether it succeeded. The record is kept for each request individually, so a single response can be traced to the exact router, model, prompt revision and API key behind it, and the same records add up to usage, latency, error and cost figures for the organization.

What is recorded

Each request record holds the following fields:

GroupField
RequestRequest id
RequestTime
RequestOrganization
RequestAPI key id
RequestCaller's customer id
RequestCaller's session id
RequestCaller's correlation id
RequestPeer IP
RoutingAlias used
RoutingRouter
RoutingModel requested
RoutingModel served
RoutingProvider, variant and region
RoutingProvider's model id
PromptBound prompt and the revision or commit used
OutcomeStatus code
OutcomeError kind and error message
OutcomeAttempts
TokensInput and output tokens
TokensCached and cache-write tokens
TokensRequest and response sizes
CostTotal cost and its parts
LatencyTotal time and time to first token
LatencyTime at the provider and time spent in Multi-Router
Machine identityIssuer, subject and machine binding of a Machine Identity workload

Routers can also store the full request and response, called payloads, for some or all requests. See Request Logging.

Where to see it

Request records are shown on several dashboard pages, each for a different purpose:

PlaceShows
TrafficEvery request, with filters, views, stats, charts and CSV export.
Request DetailOne request in full, opened from Traffic, with its payloads when they were stored.
Routers pageRequests, errors and p95 latency over the last 24 hours for the organization, and requests over the last 24 hours for each router.
Budget usageSpend over time, by router and by model, for each budget. See Budgets & Spend Control.

What decides what is recorded

Each router's Request Logging settings decide how much it records: full request metadata, metadata without fields that identify end users, failed requests only, or nothing, and whether payloads are stored. An administrator's Storage Policy sets the most any router can record. See Request Logging.

Cost

Multi-Router prices each request when it completes, from the price of the model offering that served it: a catalog offering, or a custom offering with its own prices. The cost is stored with the request, so totals in Traffic and budgets reflect the prices at the time of each request. See Cost.

Tracing a request

Every response carries an x-mr-request-id header with the request's id, which finds the request in Traffic. Callers can also send their own session and correlation ids, which Traffic records and searches, to tie requests to their own logs. See Request Tracing.

The other pages in Observability cover each part in detail:

PageCovers
TrafficViews, filters, columns, stats, charts and export.
Request DetailEvery field of a request, errors, and payloads.
Request LoggingWhat each router records, and the Storage Policy.
CostHow each request is priced.
Request TracingRequest, session and correlation ids, and response headers.
CommandsTraffic
Was this helpful?