Self-Hosting
RequirementsEnterprise
The infrastructure, addresses and network access a self-hosted installation needs.
A self-hosted installation needs somewhere to run the server, a database, a secret store, a load balancer with TLS, and three addresses for the API, the dashboard and the Portal. This page lists each requirement, so they can be prepared before installation.
The server
The Multi-Router server is a single program for Linux, provided under the enterprise agreement, that listens on port 8080. It runs anywhere a container runs, such as Kubernetes, Amazon ECS or a virtual machine with a process manager. It answers health checks at /health, which returns 200 while the process runs, and /health/ready, which returns 200 once the installation is complete and ready to serve.
Database
The database holds everything except request records and payloads. Two databases are supported:
| Database | Notes |
|---|---|
| PostgreSQL | The database is created ahead of installation, named for the installation's environment and stack joined by an underscore, such as prod_main. The wizard asks for the server address and adds this name. The server uses up to 20 connections per copy. |
| Amazon DynamoDB | Tables are created with the prefix <env>-<stack>- in the chosen region. |
The server creates and upgrades the tables it needs when it starts.
Secret store
The secret store holds the values the server needs before it can reach the database: the database connection and the encryption keys. Two secret stores are supported:
| Secret store | Suited to |
|---|---|
| AWS Secrets Manager | Installations in AWS, and any installation with several copies of the server. |
| A file | A single copy of the server, with the file on a protected volume. |
Addresses and TLS
A self-hosted installation serves three surfaces, each at its own https address, such as these, with DNS records pointing at the load balancer:
| Surface | Example address | Serves |
|---|---|---|
| Dashboard | https://dashboard.ai.example.com | The dashboard, sign-in and Admin pages. |
| API | https://api.ai.example.com | The /v1 API, the proxy endpoints including /bedrock, /vertex and /v1beta, and the SCIM endpoint. |
| Portal | https://portal.ai.example.com | The Portal, for self-service provisioning. |
The load balancer terminates TLS, forwards requests to port 8080, and appends the caller's address to X-Forwarded-For, which the server uses for rate limits. The dashboard and Portal addresses share a parent domain, which the session cookie is scoped to; the API address uses API keys.
Network access
The server reaches these services, depending on the features in use:
| Destination | Used for |
|---|---|
| Each model provider's API | Serving requests. |
| dns.google over https | Verifying the email domain of an SSO connection. |
| The identity provider's discovery and key addresses | Single sign-on and machine identity. |
| github.com | GitHub sign-in, and connecting GitHub accounts for prompts. |
| api.github.com | Reading GitHub prompts. |
| Google sign-in | Signing in with Google. |
| models.multi-router.ai, or the catalog address set on the Catalog page | Model catalog updates, when turned on. |
| The database | PostgreSQL, or the DynamoDB endpoint for the region. |
| AWS Secrets Manager | The secret store, when used. |
| The SMTP server | Email, when configured. |
| ClickHouse | Traffic, when configured. |
| Amazon S3 | Payloads, when configured. |
| Webhook endpoints | Budget alerts, over https. |
| Custom model endpoints | Custom models, over https. |
The identity provider reaches the API address for user provisioning, and GitHub reaches the dashboard address for GitHub prompts. Identity providers for single sign-on and machine identity are reached at public https addresses.