LLM API Keys in CI and Production Workloads Considered Harmful

CI pipelines, servers and Kubernetes workloads that call LLMs usually authenticate with a static API key. If that key leaks, anyone who has it can use it until someone finds the leak and replaces it.

Rahul Singh, Founder, Multi-Router 6 min read

Most workloads call LLMs with a static API key

It is almost 2027, and AI now runs well beyond chat sessions. Build pipelines summarize release notes, review code and write tests. Backend services classify support tickets and draft replies. Batch jobs label data overnight. Each of these workloads calls a model, and each one needs a credential to do it.

The usual credential is an API key stored in a CI secret, an environment variable or a Kubernetes secret. The key never expires and works from anywhere. Over time it gets copied into build logs, container images, local configuration files and other teams' pipelines. When a key leaks, it stays valid until someone notices, and replacing it means updating every system that uses it at the same time.

Most workloads already have an identity

The platforms these workloads run on already prove who they are. GitHub Actions, GitLab and CircleCI issue each pipeline run a signed OpenID Connect token that names the repository and the branch or environment it ran for. Kubernetes clusters on Amazon EKS, Google GKE and Azure AKS issue each pod a signed token for its service account. Google Cloud does the same for its workloads.

These tokens are short-lived, often only minutes, and the platform issues a new one for every run. Nothing has to be stored, and a token that is copied somewhere stops working on its own.

Multi-Router accepts that identity in place of an API key

With machine identity, a workload sends its platform token to Multi-Router in the same header it would use for an API key. Multi-Router verifies the token against the platform that issued it, finds the binding that allows that workload, and routes the request. The workload stores no Multi-Router secret.

An account admin sets this up once for each platform by registering it as an issuer. Multi-Router reads the platform's public configuration and signing keys, recognizes the kind of platform from its address, and generates an audience value. Workloads request their tokens for that audience, so a token issued for another service cannot be used with Multi-Router.

example-co · acc_7Tq2Wd9KxP

Account Settings

Machine identity

Let applications authenticate with tokens their own platform issues, instead of an API key.

Register issuer
Issuers2Bindings0
GitHub ActionsHealthyhttps://token.actions.githubusercontent.com
Token audience
mr-aud_Qx7TmVn2Lr8KpWd4YhBc3sFz9JgNe6Ua1Ro5Ti0XkCopy

Set this as the token audience in the CI workflow config. Tokens from this issuer are only accepted if they carry it.

Issuer type
GitHub Actions
If the issuer cannot be reached
Keep serving with cached keys
Maximum token lifetime
7,200 seconds

Authorized organizations

Only authorized organizations can create bindings on this issuer.

Engineeringorg_3Fh8Wq2LdK

Bindings using this issuer

3 bindings across 1 organization. Read-only — bindings are created and deleted inside each organization.

repo:example-co/billing:ref:refs/heads/mainEngineering · mib_7Hn2Qw5RtX
repo:example-co/billing:environment:productionEngineering · mib_K4vB9mZp3L
repo:example-co/docs-siteEngineering · mib_P2cW6yNs8D
Delete issuer Edit
Production clusterHealthyhttps://oidc.eks.us-east-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B71EXAMPLE1 authorized organization · 2 bindings
Account Settings, Machine identity: GitHub Actions registered as an issuer, recognized by its address, with its token audience and the bindings that use it.

Bindings decide which workloads can use which routers

An organization admin decides which workloads may call which routers by adding bindings. A binding names the issuer, the subject and the routers. The subject is the workload's identity as the platform states it, for example:

Subject Covers
repo:example-co/billing:ref:refs/heads/main Runs of the billing repository on its main branch
repo:example-co/billing:environment:production Jobs in the repository's production environment
system:serviceaccount:payments:api The api service account in the payments namespace
system:serviceaccount:payments (prefix) Every service account in the payments namespace

A binding can match one exact subject, or every subject under a prefix, so a new service in a namespace can use the namespace's routers without a new binding. When several bindings match, the most specific one applies. A binding gives a workload access to inference and, if chosen, to the list of available models.

Engineering · org_3Fh8Wq2LdK

Org Settings

Machine identity bindings

Which callers may reach which routers in Engineering, and what they are allowed to do.

Callers connect to https://api.multi-router.ai Copy

New binding

Identify the caller, then choose what it may reach.

Issuer
Token audience
mr-aud_Qx7TmVn2Lr8KpWd4YhBc3sFz9JgNe6Ua1Ro5Ti0XkCopy

Set this as the token audience in the CI workflow config. Tokens from this issuer are only accepted if they carry it.

Routers this binding may use
Release notesanthropic / anthropic/claude-sonnet-5 · rt_8Mc5Jx3VbN
Add router

With one router no routing header is sent — just a base URL and a credential.

What this binding allows
CancelCreate binding
repo:example-co/billing:ref:refs/heads/mainexactciGitHub Actions · 1 router · inference
repo:example-co/docs-siteprefixciGitHub Actions · 2 routers · inference
Org Settings, Machine identity bindings: a new binding that lets the billing repository's production environment use one router.

Every request is verified before it is routed

On every request, Multi-Router checks the token's signature against the issuer's published keys, and checks the issuer, the audience, the expiry and how long the token was issued for. Each issuer has a maximum token lifetime, two hours by default, and longer-lived tokens are refused. Multi-Router refreshes each issuer's signing keys every ten minutes and fetches new keys as soon as a token names one it has not seen.

Disabling a binding or an issuer stops its workloads on every Multi-Router server within about a minute. Administrators also choose what happens if an issuer's key endpoint cannot be reached: keep accepting tokens signed with the keys already fetched, or refuse requests until the issuer is reachable again.

  1. 1 Platform Issues a short-lived token GitHub Actions or the cluster signs a token naming the workload, for the Multi-Router audience.
  2. 2 Workload Sends it as the bearer token The same Authorization header an API key would use. Nothing is stored.
  3. 3 Multi-Router Verifies the token Signature against the issuer’s keys, then issuer, audience, expiry and lifetime.
  4. 4 Multi-Router Matches a binding The issuer and subject select the most specific enabled binding.
  5. 5 Multi-Router Routes the request To the binding’s router, and records the issuer, subject and binding in Traffic.
A request from a workload with machine identity. Every request is verified before it is routed.

A GitHub Actions workflow needs one extra step

A workflow asks GitHub for a token with the issuer's audience and sends it as a bearer token. The workflow has no stored Multi-Router credential:

permissions:
  id-token: write
  contents: read

jobs:
  summarize:
    runs-on: ubuntu-latest
    steps:
      - name: Get a Multi-Router token
        run: |
          TOKEN=$(curl -s -H "Authorization: Bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" \
            "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=$MR_AUDIENCE" | jq -r .value)
          echo "::add-mask::$TOKEN"
          echo "MR_TOKEN=$TOKEN" >> "$GITHUB_ENV"
      - name: Summarize the release
        run: |
          curl https://api.multi-router.ai/v1/chat/completions \
            -H "Authorization: Bearer $MR_TOKEN" \
            -H "Content-Type: application/json" \
            -d '{ "messages": [{ "role": "user", "content": "Summarize the changes in this release." }] }'

When the binding has one router, the request needs no model or router name.

A Kubernetes pod reads its token from a file

A pod mounts a projected service account token for the issuer's audience. The cluster writes the token to a file and replaces it before it expires, and the application reads the file for each request:

volumes:
  - name: mr-token
    projected:
      sources:
        - serviceAccountToken:
            path: token
            audience: <issuer audience>
            expirationSeconds: 3600
curl $MR_BASE_URL/chat/completions \
  -H "Authorization: Bearer $(cat /var/run/secrets/multi-router/token)" \
  -H "Content-Type: application/json" \
  -d '{ "messages": [{ "role": "user", "content": "Classify this payment dispute." }] }'

With the OpenAI SDK, the base URL points to Multi-Router and the token file's contents are passed as the API key.

Traffic shows which workload made each request

Every request records the issuer, the subject and the binding that allowed it. Traffic can be filtered by each of them and can show them as columns, so a team can see which repository, pipeline or service account made a request, which model served it and what it cost.

req_3Vk8Np2HsQ200anthropic/claude-sonnet-5anthropic
Full Request Payload →
Latency
6,148ms
Cost
$0.02
Tokens
6,200 → 900
TTFT
1,302ms
Attempts
1
Router overhead
1.2 ms
Request
Request ID
req_3Vk8Np2HsQ
Timestamp
2026-11-12 09:14:52.087
Organization
org_3Fh8Wq2LdK
API key
—
Identity type
CI
Machine issuer
mii_6Gd2Tz8QwM
Machine subject
repo:example-co/billing:environment:production
Machine binding
mib_K4vB9mZp3L
Correlation ID
—
Customer
—
Session
—
Peer IP
198.51.100.37
Routing
Alias
—
Router ID
rt_8Mc5Jx3VbN
Model requested
—
Model served
anthropic/claude-sonnet-5
Vendor
anthropic
Variant
standard
Region
global
Wire id
claude-sonnet-5
Prompt binding
Prompt ID
—
File path
—
Revision
—
A request from the billing repository's production deployment in Traffic: no API key, with the verified issuer, subject and binding.

Workloads can call LLMs without storing a secret

Machine identity lets CI pipelines and Kubernetes workloads call Multi-Router with the short-lived identity their platform already gives them. There is no long-lived key to store, leak or replace, every request is tied to the workload that made it, and admins control access through bindings. The Machine identity docs cover setup for GitHub Actions, Kubernetes and the other supported platforms.

Start for free

Create a router, set your SDK base URL to Multi-Router and see the cost of every request.