multi-router.aidocs

Self-Hosting

RequirementsEnterprise

The infrastructure, addresses and network access a self-hosted installation needs.

A self-hosted installation needs somewhere to run the server, a database, a secret store, a load balancer with TLS, and three addresses for the API, the dashboard and the Portal. This page lists each requirement, so they can be prepared before installation.

The server

The Multi-Router server is a single program for Linux, provided under the enterprise agreement, that listens on port 8080. It runs anywhere a container runs, such as Kubernetes, Amazon ECS or a virtual machine with a process manager. It answers health checks at /health, which returns 200 while the process runs, and /health/ready, which returns 200 once the installation is complete and ready to serve.

Database

The database holds everything except request records and payloads. Two databases are supported:

DatabaseNotes
PostgreSQLThe database is created ahead of installation, named for the installation's environment and stack joined by an underscore, such as prod_main. The wizard asks for the server address and adds this name. The server uses up to 20 connections per copy.
Amazon DynamoDBTables are created with the prefix <env>-<stack>- in the chosen region.

The server creates and upgrades the tables it needs when it starts.

Secret store

The secret store holds the values the server needs before it can reach the database: the database connection and the encryption keys. Two secret stores are supported:

Secret storeSuited to
AWS Secrets ManagerInstallations in AWS, and any installation with several copies of the server.
A fileA single copy of the server, with the file on a protected volume.

Addresses and TLS

A self-hosted installation serves three surfaces, each at its own https address, such as these, with DNS records pointing at the load balancer:

SurfaceExample addressServes
Dashboardhttps://dashboard.ai.example.comThe dashboard, sign-in and Admin pages.
APIhttps://api.ai.example.comThe /v1 API, the proxy endpoints including /bedrock, /vertex and /v1beta, and the SCIM endpoint.
Portalhttps://portal.ai.example.comThe Portal, for self-service provisioning.

The load balancer terminates TLS, forwards requests to port 8080, and appends the caller's address to X-Forwarded-For, which the server uses for rate limits. The dashboard and Portal addresses share a parent domain, which the session cookie is scoped to; the API address uses API keys.

Network access

The server reaches these services, depending on the features in use:

DestinationUsed for
Each model provider's APIServing requests.
dns.google over httpsVerifying the email domain of an SSO connection.
The identity provider's discovery and key addressesSingle sign-on and machine identity.
github.comGitHub sign-in, and connecting GitHub accounts for prompts.
api.github.comReading GitHub prompts.
Google sign-inSigning in with Google.
models.multi-router.ai, or the catalog address set on the Catalog pageModel catalog updates, when turned on.
The databasePostgreSQL, or the DynamoDB endpoint for the region.
AWS Secrets ManagerThe secret store, when used.
The SMTP serverEmail, when configured.
ClickHouseTraffic, when configured.
Amazon S3Payloads, when configured.
Webhook endpointsBudget alerts, over https.
Custom model endpointsCustom models, over https.

The identity provider reaches the API address for user provisioning, and GitHub reaches the dashboard address for GitHub prompts. Identity providers for single sign-on and machine identity are reached at public https addresses.

OverviewInstallation
Was this helpful?