Self-Hosting
OperationsEnterprise
Upgrading, scaling, backing up and monitoring a self-hosted installation.
Operating a self-hosted installation means keeping the server current, running enough copies for the traffic, backing up what the installation cannot rebuild, and watching its logs. This page covers each task.
Upgrading
An upgrade replaces the server with the new version and restarts it. The server applies database changes when it starts, one copy at a time on PostgreSQL, and on SIGTERM finishes requests in progress for up to 25 seconds before stopping. With several copies behind a load balancer, replacing them one at a time keeps the installation serving.
Scaling
The server keeps its data in the database, so adding copies behind the load balancer adds capacity. Rate limits, such as 300 inference requests per minute per API key, are counted within each copy. Installations with several copies use AWS Secrets Manager for secrets, and S3 or a shared directory for payloads.
Backups
A complete backup covers these parts:
| Part | Holds |
|---|---|
| The database | Accounts, people, routers, keys, prompts, policies, budgets and settings. |
| The secret store | The database connection, the encryption keys and the session signing secret. |
| Payload storage | Stored request and response payloads. |
| ClickHouse | Request records for Traffic. |
Restoring the database also needs the secret store from the same time, so the two are backed up together.
Logs
The server writes its logs to standard output, with each request's id, the same id returned in the x-mr-request-id header. The level is set with the RUST_LOG environment variable, at info by default.
The model catalog
The model catalog, with its models, offerings and prices, ships inside the server and updates with each upgrade. With Catalog fetching turned on, the server also fetches new catalog versions from models.multi-router.ai, or the address set on the Catalog page, applied automatically or after approval on the Catalog page. Models the catalog lacks can be added as custom models. See Model Catalog and Custom Models & Pricing.