AI

Bifrost LLM Gateway: Setup, Virtual Keys, Fallbacks and Web UI

You’ve got Ollama on one box and vLLM on another. Every app, script and coding agent on the team carries its own base URL and its own idea of which model to call. The Bifrost LLM gateway sits between them: clients get one OpenAI-compatible endpoint and a virtual key, and Bifrost decides which backend answers, what each key may spend, when to fail over and when to serve a cached reply instead. LiteLLM is the alternative gateway most teams weigh against it, and our LiteLLM proxy guide puts that one in front of the same Ollama and vLLM backends.

Original content from computingforgeeks.com - post 183770

Almost every Bifrost walkthrough online comes from Maxim, the company that builds it, so we ran the open source image ourselves and wrote down where it surprised us. This guide covers the Docker setup, the web UI on port 8080, virtual keys with budgets and rate limits, weighted routing and failover between Ollama and vLLM, direct and semantic caching, the MCP gateway, Prometheus metrics and Claude Code. Each feature is configured in the web UI or config.json and checked with a real request. We confirmed the whole setup working with Bifrost v2.4.0 on Ubuntu 24.04 in November 2026, on a 4 vCPU lab VM with no GPU.

Plan the request path

Everything in this guide runs on one host we called broker01, in a single Docker network. Apps and coding agents send requests to Bifrost with a virtual key, Bifrost checks the semantic cache in Redis, then picks Ollama or vLLM by weight and falls back to the other when one fails. The MCP server is optional and only shows up for keys that are allowed to use it. Because every backend sits on the Docker network, none of them needs a published port, which matters later when Bifrost’s private network guard gets in the way.

Bifrost LLM gateway architecture: apps and coding agents send virtual-key requests to Bifrost, which uses Redis for the semantic cache and routes to Ollama, vLLM and an MCP server

Both backends serve the same model, Qwen2.5 0.5B Instruct, under the same name qwen2.5:0.5b. That is deliberate: weighted routing and automatic failover only make sense when either backend can answer the same model name. On real hardware you’d run vLLM on a GPU with a bigger model, as in our vLLM production install guide, and the gateway side of this guide stays identical.

Check what the OSS image includes

Bifrost’s docs describe features without saying which ones are open source, so we opened every page of the OSS web UI and tested the gateway features with real requests. Virtual keys, budgets, rate limits, fallbacks, weighted routing, semantic caching, the MCP gateway and Prometheus metrics all work in the free maximhq/bifrost image. Fourteen UI pages show the same banner instead of a form: “This feature is a part of the Bifrost enterprise license.” The table separates what we tested from what we only read in the docs.

FeatureOSS imageHow we know
Virtual keys, budgets, rate limitsYesLab: 429 and 402 responses below
Weighted routing and fallbacksYesLab: killed backend, injected 429s
Direct and semantic cachingYesLab: Redis Stack plus Ollama embeddings
MCP gateway with per-key tool scopingYesLab: tools/list and tools/call on /mcp
Prometheus /metrics, LLM logs, dashboardYesLab: scraped with the admin login
Teams and Customers pagesTeams and Customers pages open in the OSS UI; not tested here. The docs list the wider hierarchy (users, business units, RBAC) as Enterprise.UI probe, not tested; docs for the Enterprise part
OpenTelemetry and other exports (Datadog, BigQuery, Kafka, Pub/Sub, Splunk, Maxim)Page opens, one tile marked Coming soonUI probe, page opens, export not run
SCIM, RBAC, user governanceEnterprise bannerUI probe, gated
Business Units, Audit LogsEnterprise bannerUI probe
Guardrails, Circuit Breaker, Adaptive RoutingEnterprise bannerUI probe
Cluster mode (high-availability clustering), Alerting channels, Edge ControlEnterprise bannerUI probe; the enterprise overview lists high-availability clustering as Enterprise
SSO / OIDC login (Okta, Microsoft Entra, Keycloak, Zitadel, Google Workspace)EnterpriseDocs only (enterprise overview), not run in the lab
Log exports to S3, GCS, BigQuery; Datadog connectorEnterpriseDocs only (enterprise overview)

One gap worth planning around: Circuit Breaker is Enterprise, yet plain fallbacks are not. The free build retries and falls over on every failed request, but there’s no circuit breaker that stops sending traffic to a backend it already knows is dead, so a dead primary costs each request a failed attempt first. The Observability page is the opposite case: it opens without a banner and lists OpenTelemetry next to Prometheus and the commercial sinks, but we only ran the Prometheus scrape, and the enterprise overview still names log exports and the Datadog connector as paid features, so treat the other connectors as unproven until you push a trace through one. The MCP gateway is the reverse: the project README lists it among the capabilities enterprise deployments unlock, yet the OSS v2.4.0 image answered tools/list and tools/call on /mcp for a scoped virtual key in our lab. The enterprise overview narrows the paid part to Virtual MCPs, curated tool bundles served at /mcp/<slug>; that page opened in the OSS UI, but we did not create one. The repository itself is Apache-2.0, and the gated pages ship in the same image as locked screens rather than as a separate download. LiteLLM draws its own line between free and paid features, and the guide to our self-hosted LiteLLM proxy runs the same feature check on its side, then points back to this Bifrost gateway guide for readers weighing the two.

Bifrost open source build showing the Guardrails page locked behind the enterprise license

Guardrails is the gated page most people go looking for first, because PII masking is a common reason to put a gateway in the path at all. If you need it today on the free tier, it has to live in front of Bifrost or inside the app.

Prepare the host and the backends

The lab host was a 4 vCPU, 10 GB Ubuntu 24.04 VM with Docker Engine and the Compose plugin from the Ubuntu archive. Bifrost itself showed 158 MiB of RAM in docker stats at the end of the lab. The backends drive the sizing, since the CPU build of vLLM held 6.3 GiB for a 0.5B model and Ollama about 870 MiB. vLLM’s CPU image wants AVX-512 on the host CPU, and a Proxmox VM needs the host CPU type to expose it. Treat these numbers as a floor for a lab, not as production sizing.

Create a working directory and a setup token now, because Bifrost refuses to create its first admin account without one and adding it later costs a restart.

mkdir -p ~/bifrost/data && cd ~/bifrost
echo "BIFROST_SETUP_TOKEN=$(openssl rand -hex 32)" > .env && chmod 600 .env
docker volume create ollama
docker volume create hf

Two named volumes keep Ollama models and the Hugging Face cache across container rebuilds, so a docker compose down does not mean downloading the models again. Now create the Compose file.

vim ~/bifrost/compose.yaml

Paste the stack we ran. The vLLM service uses the CPU image and serves the model under the same name Ollama uses, which is what lets one model name route to either backend:

services:
  bifrost:
    image: maximhq/bifrost:v2.4.0
    container_name: bifrost
    ports:
      - "8080:8080"
    environment:
      BIFROST_SETUP_TOKEN: ${BIFROST_SETUP_TOKEN}
    volumes:
      - ./data:/app/data
    restart: unless-stopped

  ollama:
    image: ollama/ollama:0.40.14
    container_name: ollama
    volumes:
      - ollama:/root/.ollama
    restart: unless-stopped

  vllm:
    image: vllm/vllm-openai-cpu:v0.33.0
    container_name: vllm
    shm_size: 2g
    environment:
      VLLM_CPU_KVCACHE_SPACE: "2"
    command: ["--model", "Qwen/Qwen2.5-0.5B-Instruct", "--served-model-name", "qwen2.5:0.5b", "--max-model-len", "4096"]
    volumes:
      - hf:/root/.cache/huggingface
    restart: unless-stopped

  redis:
    image: redis/redis-stack-server:7.4.0-v8
    container_name: redis
    restart: unless-stopped

volumes:
  ollama:
    external: true
  hf:
    external: true

Only Bifrost publishes a port. Redis is the Stack build rather than plain Redis because the semantic cache needs vector search, and the image tags are pinned so the stack you build matches the one we tested (Bifrost v2.4.0 release notes).

Start Bifrost with Docker Compose

Bifrost reads /app/data/config.json at startup when it exists and seeds its SQLite config store from it. We start with Ollama only and add vLLM through the web UI, so you see both ways of configuring a provider. Create the file.

vim ~/bifrost/data/config.json

Add the client settings and the Ollama provider. Ollama is a native provider type, so the backend URL goes in ollama_key_config on the key, and allow_private_network is required because the container answers on a private Docker address:

{
  "$schema": "https://www.getbifrost.ai/schema",
  "client": {
    "enforce_auth_on_inference": true,
    "enable_logging": true
  },
  "providers": {
    "ollama": {
      "keys": [
        {
          "name": "ollama-local",
          "value": "",
          "models": ["*"],
          "weight": 1.0,
          "ollama_key_config": { "url": "http://ollama:11434" }
        }
      ],
      "network_config": {
        "allow_private_network": true,
        "default_request_timeout_in_seconds": 120
      }
    }
  }
}

The Bifrost container runs as UID 1000 with group 0 and refuses to start when it cannot write its data directory. On our host the login user was UID 1001, the container went into a restart loop, and the fix is to hand the directory to that UID before the first start:

sudo chown -R 1000:0 ~/bifrost/data
sudo chmod 770 ~/bifrost/data
docker compose up -d

Pull the chat model and the embedding model into Ollama. The embedding model is only needed for semantic caching, but pulling it now keeps the later section short:

docker exec ollama ollama pull qwen2.5:0.5b
docker exec ollama ollama pull nomic-embed-text

Give the gateway time before you test it. The first start of this stack took 42 seconds, almost all of it after the log line “initializing model catalog”, and until then curl got “Connection reset by peer” instead of an answer. Later restarts took 12 seconds and then under half a second. Once the health check passes, all four containers should be up:

curl -s localhost:8080/health; echo
docker compose ps --format "table {{.Name}}\t{{.Image}}\t{{.Status}}"

Look for the database ping in the health response, then check that Compose shows Bifrost as healthy next to the three backends:

{"components":{"db_pings":"ok"},"status":"ok"}
NAME      IMAGE                               STATUS
bifrost   maximhq/bifrost:v2.4.0              Up 6 minutes (healthy)
ollama    ollama/ollama:0.40.14               Up 10 minutes
redis     redis/redis-stack-server:7.4.0-v8   Up 33 minutes
vllm      vllm/vllm-openai-cpu:v0.33.0        Up 33 minutes

Now send a chat request without any key. The config.json above sets enforce_auth_on_inference to true, so this should fail rather than reach Ollama. Leave that line out and Bifrost accepts anonymous requests: a fresh container with no config.json passed our keyless request straight through to provider lookup.

curl -s localhost:8080/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"model":"ollama/qwen2.5:0.5b","messages":[{"role":"user","content":"Say hi in three words."}]}' | jq .

A 401 is the behaviour you want on a shared gateway. Note the top-level type field: client code should branch on it, because the same status code covers several different refusals:

{
  "type": "virtual_key_required",
  "is_bifrost_error": false,
  "status_code": 401,
  "error": {
    "message": "virtual key is required. Provide a virtual key via the x-bf-vk header."
  },
  "extra_fields": {
    "routing_info": {},
    "provider": "ollama",
    "original_model_requested": "qwen2.5:0.5b",
    "resolved_model_used": "qwen2.5:0.5b",
    "request_type": "chat_completion"
  }
}

Ollama is configured and protected. Before you can create a key, though, you need to get into the admin side, which is locked as well.

Create the admin account with the setup token

This release locks the whole /api surface, and even /metrics, until an admin account exists. If you start it without a setup token, the API answers 403 with “dashboard auth is not configured and no setup token is set”, and the only fix is a restart with BIFROST_SETUP_TOKEN set. With the token from your .env in place, open http://10.0.1.50:8080 (your host’s address) and follow these steps:

  1. Paste the value of BIFROST_SETUP_TOKEN from ~/bifrost/.env into Setup token.
  2. Click Continue. Bifrost opens the Security Settings page.
Bifrost Finish setting up screen asking for the setup token before the admin account exists
  1. 1Setup token
  2. 2Continue

On the security page, turn on Password protect the dashboard, enter admin as the Username (the API commands below assume it) and a Password of at least 12 characters with upper and lower case, a digit and a symbol, and leave Enforce Virtual Keys on Inference on. Click Save Changes. The page says what happens next: saving creates the admin account, after which the setup token stops working, so there’s nothing to rotate afterwards.

Bifrost Security Settings with Password protect the dashboard on, admin username and password, and Enforce Virtual Keys on Inference
  1. 1Password protect the dashboard
  2. 2Username
  3. 3Password
  4. 4Enforce Virtual Keys on Inference
  5. 5Save Changes

Those credentials also work as HTTP Basic auth on the admin API and on /metrics, which is how the commands later in this guide call the API. Export them in the shell you use for testing, along with the virtual key you create in the next section:

export BIFROST_ADMIN_PASS='your-admin-password'
export BIFROST_VK='sk-bf-paste-the-team-a-key-here'

You set BIFROST_VK once team-a exists. Everything after this point uses those two variables, plus VK_ID from the budgets section, so a new terminal needs all three set again.

Add vLLM in the web UI

Bifrost has a native vLLM provider, so you do not need the generic custom provider for it. In the web UI, go to Models and open Model Providers, click Add New Provider and choose vLLM from the list. The provider appears with no models until you add one.

Bifrost Model Providers page with Add New Provider open and vLLM in the provider list
  1. 1Add New Provider
  2. 2vLLM

Click Add Model on the empty vLLM provider and fill in the dialog. The values must match what the vLLM container serves, because Bifrost sends the model name through unchanged:

  1. Name: vllm-cpu, a label for this backend in logs and routing info.
  2. Server URL: http://vllm:8000, the Compose service name and port.
  3. Model Name: qwen2.5:0.5b, the --served-model-name from compose.yaml.
  4. Click Save.
Bifrost vLLM Add new model dialog with the name, server URL http://vllm:8000 and model name qwen2.5:0.5b
  1. 1Name
  2. 2Server URL
  3. 3Model Name
  4. 4Save

Our first vLLM request through the gateway failed with a 502 even though the container was healthy, and the model row showed a red status icon. The response named the cause: "connection to private IP 172.19.0.5 is not allowed". A provider created in the UI does not inherit allow_private_network, and Bifrost blocks private addresses by default as SSRF protection. Click Edit Provider Config, stay on the Network tab, turn on Allow Private Network and click Save Network Configuration.

Bifrost vLLM provider configuration with Allow Private Network switched on so it can reach a container on the Docker network
  1. 1Network
  2. 2Allow Private Network
  3. 3Save Network Configuration

No restart is needed for the switch. Cloud metadata addresses in 169.254.0.0/16 stay blocked even with it on, so turning it on for a LAN backend doesn’t open the metadata endpoint to anyone holding a key.

Bifrost Model Providers page listing flaky, Ollama and vLLM, with the vllm-cpu entry healthy after Allow Private Network

Come back to this test after you create the team-a key in the next section. With that key in hand, the same question sent to both backends by their full provider/model names proves the wiring end to end:

for m in ollama/qwen2.5:0.5b vllm/qwen2.5:0.5b; do
  curl -s localhost:8080/v1/chat/completions -H "Authorization: Bearer $BIFROST_VK" -H "Content-Type: application/json" \
    -d "{\"model\":\"$m\",\"messages\":[{\"role\":\"user\",\"content\":\"Name one Linux init system in one word.\"}],\"max_tokens\":10}" \
    | jq -c '{provider: .extra_fields.provider, model, content: .choices[0].message.content, tokens: .usage.total_tokens, latency_ms: .extra_fields.latency}'
done

Both providers answer, and extra_fields.provider tells you which one did. The 683 ms on vLLM against 59 ms on Ollama is a CPU-only lab with a warm Ollama, not a statement about either engine; our Ollama vs vLLM vs llama.cpp benchmark compares them properly on a GPU:

{"provider":"ollama","model":"qwen2.5:0.5b","content":"Linux.","tokens":41,"latency_ms":59}
{"provider":"vllm","model":"qwen2.5:0.5b","content":"init","tokens":40,"latency_ms":683}

A 0.5B model will not win quiz nights, but the routing is what we’re testing here, and both backends now answer the same model name.

Create virtual keys with budgets and limits

A virtual key is what you hand to an app or a person instead of a backend URL. It decides which providers and models the caller may use, how many requests or tokens it gets per window, and how much it may spend. Go to Governance, open Virtual Keys and click Add Virtual Key. Name it team-a, then pick Ollama and vLLM in the provider selector so both land under Provider Configurations. A key with no providers listed can reach nothing, because the list is deny by default.

Bifrost Create Virtual Key dialog with Ollama and vLLM added under Provider Configurations
  1. 1Ollama
  2. 2vLLM

Scroll to Rate Limiting Configuration, set Maximum Requests to 20 with the reset period Every Minute, and click Create. Twenty a minute is low on purpose: it lets you see the limit trip in a few seconds of testing, and you can raise it afterwards without reissuing the key.

Bifrost Create Virtual Key dialog with a 20 requests per minute rate limit and the Create button
  1. 1Maximum Requests
  2. 2Every Minute
  3. 3Create

The key list shows the value masked as sk-bf- plus a few characters, with an icon to copy it. Copy it into BIFROST_VK. The second key in the list, ci-bot, comes from config.json, which is covered at the end of this guide; the two coexist because Bifrost merges file and UI configuration per entity.

Bifrost Virtual Keys list with team-a from the web UI and ci-bot from config.json, showing budgets and rate limits

Once team-a has made 20 requests inside a minute, the next one comes back as a 429. The headers carry the provider Bifrost would have used, and the body states the limit and its window, which is enough for a client to back off sensibly:

HTTP/1.1 429 Too Many Requests
Server: fasthttp
Content-Type: application/json
X-Bifrost-Provider: ollama

{"type":"request_limited","is_bifrost_error":false,"status_code":429,"error":{"message":"Rate limit exceeded: rate limit violated for vk_model_config *: [request limit exceeded (20/20, resets every 1m)]"},"extra_fields":{"routing_info":{},"provider":"ollama","original_model_requested":"qwen2.5:0.5b","resolved_model_used":"qwen2.5:0.5b","request_type":"chat_completion","error_type":"policy_rate_limited"}}

Budgets catch people out on self-hosted models. We gave team-a a budget of $0.001 a day with Add Budget in the key dialog (or budgets in the API). After five requests the usage still read 0, because Bifrost prices requests from its model catalog and a local Ollama or vLLM model has no catalog price. A budget on a local model never trips until you give the model a price. Add a pricing override through the admin API (the Pricing Overrides page under Models does the same):

curl -s -u "admin:$BIFROST_ADMIN_PASS" -X POST localhost:8080/api/governance/pricing-overrides -H "Content-Type: application/json" \
  -d '{"name":"qwen-internal-rate","scope_kind":"global","match_type":"exact","pattern":"qwen2.5:0.5b","request_types":["chat_completion"],"patch":{"input_cost_per_token":0.00001,"output_cost_per_token":0.00002}}'

Set the price to whatever a GPU hour costs you divided by the tokens it serves, so budgets become an internal chargeback. With that override in place, the first request cost about $0.0012 and every request after it was refused with a 402:

{"status":200,"type":null,"tokens":76,"cost":null,"err":null}
{"status":402,"type":"budget_exceeded","tokens":null,"cost":null,"err":"Budget exceeded: vk_model_config * budget exceeded: 0.0012 >= 0.0010 dollars"}
{"status":402,"type":"budget_exceeded","tokens":null,"cost":null,"err":"Budget exceeded: vk_model_config * budget exceeded: 0.0012 >= 0.0010 dollars"}

The check runs before the request, against usage already recorded, so one request can overshoot the cap: we ended at $0.0012 on a $0.001 budget. Raise the limits before you go on, or every later test on team-a hits this 402 or the 20-a-minute limit. This PUT sets a $5 daily budget and 200 requests a minute, and also gives the two providers the weights used in the routing section:

VK_ID=$(curl -s -u "admin:$BIFROST_ADMIN_PASS" localhost:8080/api/governance/virtual-keys | jq -r '.virtual_keys[] | select(.name=="team-a") | .id')
curl -s -u "admin:$BIFROST_ADMIN_PASS" -X PUT localhost:8080/api/governance/virtual-keys/$VK_ID -H "Content-Type: application/json" \
  -d '{"provider_configs":[{"provider":"ollama","weight":0.7,"allowed_models":["*"],"key_ids":["*"]},{"provider":"vllm","weight":0.3,"allowed_models":["*"],"key_ids":["*"]}],"budgets":[{"max_limit":5,"reset_duration":"1d"}],"rate_limit":{"request_max_limit":200,"request_reset_duration":"1m"}}' | jq -c '{b: .virtual_key.budgets[0].max_limit, r: .virtual_key.rate_limit.request_max_limit}'

It echoes the new budget and request limit back:

{"b":5,"r":200}

Model and provider scoping is enforced the same way. The ci-bot key is defined in the config.json at the end of this guide; if you apply that file now, start the flaky and everything containers from the next sections first so its provider and MCP server resolve. On Ollama it allows only qwen2.5:0.5b, it has no vLLM access, and asking for anything outside that returns a 403 with a type that says which rule refused it:

for m in ollama/llama3.2:1b vllm/qwen2.5:0.5b; do
  curl -s localhost:8080/v1/chat/completions -H "Authorization: Bearer sk-bf-ci-bot-lab-0001" -H "Content-Type: application/json" \
    -d "{\"model\":\"$m\",\"messages\":[{\"role\":\"user\",\"content\":\"ok\"}]}" | jq -c '{type, status_code, msg: .error.message}'
done

The first request names a model the key does not list, the second a provider it does not list:

{"type":"model_blocked","status_code":403,"msg":"Model 'llama3.2:1b' is not allowed for virtual key 'ci-bot'"}
{"type":"provider_blocked","status_code":403,"msg":"Provider 'vllm' is not allowed for virtual key 'ci-bot'"}

That makes four refusal types to watch for, virtual_key_required (401), request_limited (429), budget_exceeded (402) and model_blocked or provider_blocked (403). Clients that log the type field will tell you which policy hurt them without a trip to the gateway logs.

Configure weighted routing and failover

Weighted routing lives on the virtual key. Give each provider a weight and send a bare model name without the provider/ prefix; a prefixed name bypasses the weights and goes straight to that provider. The PUT in the previous section already gave team-a weights of 0.7 for Ollama and 0.3 for vLLM, and it carries the detail that cost us the most time in the lab.

Keep "key_ids":["*"] in every provider config you send. Our first PUT left it out and the API stored allow_all_keys: false with an empty key list. A few seconds later every request on team-a failed with “no keys found for provider” on both backends, while the containers were perfectly healthy. Read the key back to confirm both providers kept their weights and key access:

curl -s -u "admin:$BIFROST_ADMIN_PASS" localhost:8080/api/governance/virtual-keys | jq -c '.virtual_keys[] | select(.name=="team-a") | .provider_configs[] | {provider,weight,allow_all_keys}'

Both providers must show allow_all_keys: true:

{"provider":"ollama","weight":0.7,"allow_all_keys":true}
{"provider":"vllm","weight":0.3,"allow_all_keys":true}

Twenty requests for the bare name then split 14 to Ollama and 6 to vLLM, which is the 70/30 weighting within the noise you expect from 20 samples. Now stop Ollama and send ten more:

docker stop ollama
for i in $(seq 1 10); do
  curl -s -o /tmp/r.json -w "%{http_code} " localhost:8080/v1/chat/completions -H "Authorization: Bearer $BIFROST_VK" -H "Content-Type: application/json" \
    -d '{"model":"qwen2.5:0.5b","messages":[{"role":"user","content":"ok"}],"max_tokens":3}'
  jq -r '[.extra_fields.provider, (.extra_fields.routing_info.is_fallback // false)] | @tsv' /tmp/r.json
done | sort | uniq -c
jq '{provider: .extra_fields.provider, routing_info: .extra_fields.routing_info, content: .choices[0].message.content}' /tmp/r.json
docker start ollama

All ten came back 200 from vLLM. Eight of them had picked Ollama first and fell over, which routing_info records with is_fallback and the original primary_provider; two went to vLLM directly by weight. Bifrost builds that fallback chain from the key’s weights when the request doesn’t list fallbacks itself:

      2 200 vllm	false
      8 200 vllm	true
{
  "provider": "vllm",
  "routing_info": {
    "provider": "vllm",
    "model": "qwen2.5:0.5b",
    "key": "vllm-cpu",
    "is_fallback": true,
    "primary_provider": "ollama",
    "primary_model": "qwen2.5:0.5b",
    "requested_model": "qwen2.5:0.5b"
  },
  "content": "Hello! How can I assist you today? Feel free to ask me anything about"
}

A dead container is the easy case because the connection is refused at once. Rate limiting upstream is the harder one, so we put a 20-line Python mock on the same network that answers every chat request with an OpenAI-style 429 and a Retry-After header:

vim ~/bifrost/flaky.py

On GET the mock lists one model so Bifrost’s model refresh succeeds, and refuses every POST:

import json
from http.server import BaseHTTPRequestHandler, HTTPServer


class H(BaseHTTPRequestHandler):
    def do_POST(self):
        self.rfile.read(int(self.headers.get("Content-Length", 0)))
        body = json.dumps({"error": {"message": "Rate limit reached for requests", "type": "requests", "code": "rate_limit_exceeded"}}).encode()
        self.send_response(429)
        self.send_header("Content-Type", "application/json")
        self.send_header("Retry-After", "20")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def do_GET(self):
        body = json.dumps({"object": "list", "data": [{"id": "qwen2.5:0.5b", "object": "model"}]}).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)


HTTPServer(("0.0.0.0", 9000), H).serve_forever()

Run it on the Compose network, then register it as the custom provider flaky with max_retries: 2 (the full config.json is at the end of this guide):

docker run -d --name flaky --network bifrost_default -v ~/bifrost/flaky.py:/flaky.py:ro python:3.12-alpine python /flaky.py

The test uses the ci-bot key, which may call both flaky and Ollama. Send one request to the mock with a request-level fallbacks list naming Ollama, and the same request without the list for comparison:

curl -s -o /tmp/b.json -w "http=%{http_code} time=%{time_total}s\n" localhost:8080/v1/chat/completions -H "Authorization: Bearer sk-bf-ci-bot-lab-0001" -H "Content-Type: application/json" \
  -d '{"model":"flaky/qwen2.5:0.5b","fallbacks":["ollama/qwen2.5:0.5b"],"messages":[{"role":"user","content":"Reply with the word ready."}]}'
jq '{content: .choices[0].message.content, routing_info: .extra_fields.routing_info}' /tmp/b.json

Without fallbacks the request came back 429 after 1.7 seconds. The mock logged six POSTs for the two requests, three each: the first try plus two retries with backoff. With the list, Bifrost exhausted the same retries and then answered from Ollama in 2.8 seconds:

## no fallback
http=429 time=1.728913s
{"type":null,"status_code":429,"msg":"Rate limit reached for requests"}
## with fallbacks
http=200 time=2.816329s
{
  "content": "Ready. Ready to assist you. If you have any questions or need further clarification, feel free to ask. Ready to provide information or assistance.",
  "routing_info": {
    "provider": "ollama",
    "model": "qwen2.5:0.5b",
    "key": "ollama-local",
    "is_fallback": true,
    "primary_provider": "flaky",
    "primary_model": "qwen2.5:0.5b",
    "requested_provider": "flaky",
    "requested_model": "qwen2.5:0.5b"
  }
}

That 1.7 seconds is the price of retries on a provider that isn’t going to recover within the backoff window, and without the Enterprise circuit breaker every request pays it. Keep max_retries at 0 or 1 for providers you also list as fallbacks, and save the higher counts for a single provider with no alternative. Note that the upstream 429 has a null type; only Bifrost’s own policy refusals fill that field.

Turn on direct and semantic caching

The semantic_cache plugin answers repeated questions without touching a backend, and it has two ways of deciding that a question is a repeat.

  • A direct hit is an exact hash match on the request, which costs almost nothing.
  • A semantic hit embeds the prompt, compares it with stored prompts and replays the answer of one above the similarity threshold.

Both modes need a vector store, even direct-only, and the semantic side also needs an embedding model. We used Redis Stack and nomic-embed-text on the same Ollama. Add these blocks to config.json and restart Bifrost:

  "vector_store": {
    "enabled": true,
    "type": "redis",
    "config": {
      "addr": "redis:6379"
    }
  },
  "plugins": [
    {
      "enabled": true,
      "name": "semantic_cache",
      "config": {
        "provider": "ollama",
        "embedding_model": "nomic-embed-text",
        "dimension": 768,
        "threshold": 0.85,
        "ttl": "10m",
        "cache_by_model": true,
        "cache_by_provider": true,
        "default_cache_key": "lab"
      }
    }
  ],

Keep default_cache_key in the file. Without it a request only touches the cache when the client sends an x-bf-cache-key header, which is easy to forget and makes the cache look broken. The Bifrost log confirms the plugin loaded with a plugin status line for semantic_cache. We then asked the same question twice, reworded it once, and asked something unrelated:

## 1 first ask
time=4.604431s {"hit":false,"type":null,"sim":null}
## 2 exact repeat
time=0.002425s {"hit":true,"type":"direct","sim":null}
X-Bifrost-Cache-Hit-Type: direct
## 3 reworded
time=0.263637s {"hit":true,"type":"semantic","sim":0.9456475973129}
X-Bifrost-Cache-Hit-Type: semantic
X-Bifrost-Cache-Similarity: 0.9456475973129
X-Bifrost-Cache-Threshold: 0.85
## 4 unrelated
time=5.243951s {"hit":false,"type":null,"sim":null}

An exact repeat came back in 2.4 ms against 4.6 s for the miss. “Which port does PostgreSQL use by default?” matched “What port does PostgreSQL listen on by default?” at 0.946 similarity and returned in 264 ms, most of it the embedding call. The flip side showed up later: “Count to 11” matched an earlier “Count to” prompt at 0.9999 similarity and came back as a semantic hit, so at 0.85 the cache will answer one count with another’s reply. Prompts that differ only in a number or an ID need direct-only mode or x-bf-cache-no-store: true.

Bifrost Caching settings with Direct plus Semantic mode, Ollama nomic-embed-text embeddings and a 0.85 similarity threshold

Settings, then Caching, shows the same plugin loaded from config.json, and changes there apply live without a restart. Cache entries are scoped per virtual key by default, so team-a never receives an answer that was cached for ci-bot.

Limit MCP tools per virtual key

Bifrost can connect to MCP servers and re-serve their tools on its own /mcp endpoint, filtered per virtual key. We ran the reference “everything” MCP server over streamable HTTP on the Compose network:

docker run -d --name everything --network bifrost_default node:22-alpine npx -y @modelcontextprotocol/server-everything streamableHttp

Then we registered it in config.json under mcp.client_configs with connection_type set to http and connection_string set to http://everything:3001/mcp. After the restart the log said Connected to MCP server 'everything' and the admin API listed 13 tools, among them get-env, which prints the server’s environment variables. That tool is exactly why per-key scoping matters. Our first tools/list call with team-a’s key returned nothing at all:

curl -s localhost:8080/mcp -H "Authorization: Bearer $BIFROST_VK" -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'

An empty list is the correct answer, because MCP access on a virtual key is deny by default just like providers:

{"jsonrpc":"2.0","id":1,"result":{"tools":[]}}

Grant team-a two harmless tools. In the UI this is the MCP Server Configurations picker on the key; through the API it’s mcp_configs, and the provider list goes in the same body so it isn’t wiped:

curl -s -u "admin:$BIFROST_ADMIN_PASS" -X PUT localhost:8080/api/governance/virtual-keys/$VK_ID -H "Content-Type: application/json" \
  -d '{"provider_configs":[{"provider":"ollama","weight":0.7,"allowed_models":["*"],"key_ids":["*"]},{"provider":"vllm","weight":0.3,"allowed_models":["*"],"key_ids":["*"]}],"mcp_configs":[{"mcp_client_name":"everything","tools_to_execute":["echo","get-sum"]}]}'

Listing again shows the two granted tools with the client name as a prefix. A tools/call for everything-get-sum works, and one for everything-get-env is refused by the gateway before it reaches the MCP server:

everything-echo
everything-get-sum
---
{"jsonrpc":"2.0","id":2,"result":{"content":[{"type":"text","text":"The sum of 40 and 2 is 42."}]}}
{"jsonrpc":"2.0","id":3,"result":{"content":[{"type":"text","text":"Tool execution failed: MCP tool 'everything-get-env' is not allowed for virtual key 'team-a'"}],"isError":true}}

Granting tools has a side effect the docs mention only in passing. Bifrost injects every tool a key may use into that key’s chat completions as well, unless the client sends its own x-bf-mcp-include-tools header. The same “Count to 5” prompt cost 638 prompt tokens on team-a against 33 on ci-bot, and the 0.5B model answered team-a with an empty string. An empty header (curl’s -H "x-bf-mcp-include-tools;") brought it back to 33 tokens. The docs also offer disable_auto_tool_inject to switch the injection off on the server side:

{"key":"sk-bf-4adf48fb","prompt_tokens":638,"finish":"stop","tool_calls":0,"content":""}
{"key":"sk-bf-ci-bot-l","prompt_tokens":33,"finish":"stop","tool_calls":0,"content":"Sure, here are the numbers from 1 to 5:\n\n1\n2\n3\n4\n5"}
{"hdr":"empty include-tools","prompt_tokens":33,"content":"Count to 5 is a simple counting activity that can "}

So give MCP tools to the keys that run agents, not to the key your chat app shares, or every chat turn pays for tool definitions it never uses. The MCP Catalog page shows the connected server, its 13 of 13 enabled tools and how many virtual keys can reach it.

Bifrost MCP Server Catalog listing the everything MCP server over HTTP with 13 of 13 tools enabled and access for one virtual key

By default, tool calls that a model returns in a chat completion aren’t executed by the gateway; only Agent Mode can auto-execute tools you list in tools_to_auto_execute. The client reviews them and calls the execute endpoint or /mcp itself, which keeps a human or the agent loop in charge of side effects.

Collect Prometheus metrics and logs

Prometheus metrics are built into the open source image through its telemetry plugin, with no extra configuration. Once the admin account exists, /metrics wants the same Basic auth as the API and returns 401 without it, so the scrape job needs basic_auth with the admin login (API-key auth for the scraper is an Enterprise feature). We pulled the request, cost and cache counters with their labels:

curl -s -u "admin:$BIFROST_ADMIN_PASS" localhost:8080/metrics | grep -E "^bifrost_(success_requests_total|error_requests_total|cache_hits_total|cost_total)" \
  | grep -o -E "^[a-z_]+|cache_type=\"[a-z]*\"|provider=\"[a-z]*\"|virtual_key_name=\"[a-z-]*\"| [0-9.e+-]+$" | paste -sd" " | sed -E "s/ (bifrost_)/\n\1/g"

Every series carries the provider and the virtual key name, which is what you need for per-team dashboards. Rows repeat because each one is a different model and method combination that the filter above hides, and we trimmed the list to the first eleven lines:

bifrost_cache_hits_total cache_type="semantic" provider="ollama" virtual_key_name="team-a"  3
bifrost_cache_hits_total cache_type="semantic" provider="ollama" virtual_key_name="team-a"  9
bifrost_cache_hits_total cache_type="semantic" provider="ollama" virtual_key_name="team-a"  2
bifrost_cost_total provider="ollama" virtual_key_name="team-a"  0.015960000000000002
bifrost_cost_total provider="ollama" virtual_key_name="ci-bot"  0.0007900000000000001
bifrost_cost_total provider="ollama" virtual_key_name="team-a"  0.0067800000000000004
bifrost_error_requests_total provider="vllm" virtual_key_name="team-a"  3
bifrost_success_requests_total provider="ollama" virtual_key_name="team-a"  3
bifrost_success_requests_total provider="ollama" virtual_key_name="team-a"  4
bifrost_success_requests_total provider="ollama" virtual_key_name="ci-bot"  1
bifrost_success_requests_total provider="ollama" virtual_key_name="team-a"  9

Counters live in memory and reset when Bifrost restarts, which our config.json edits did several times; Prometheus handles that with rate(), but a one-off curl after a restart shows less than you expect. For history, the built-in LLM Logs page keeps every request in SQLite with the type, model, provider, app and latency. Bifrost labelled the Claude Code request with its app name on its own.

Bifrost LLM Logs page with 182 requests in the last hour, including a Claude Code request routed to Ollama

Our own tests explain the 36.81% success rate in that header, since most of the lab traffic was built to fail with missing keys, rate limits, budgets, blocked models and the 429 mock. The Dashboard page rolls the same data into charts for request volume, tokens, cost and cache hit rate.

Bifrost dashboard with request volume, token usage, cost and local cache hit rate from the lab traffic

The logs store adds memory overhead, which the Logs Settings page puts at 40 to 60 MB, and it records prompt and response content unless you turn content logging off globally or per key. On a gateway that sees customer data, decide that before the first real request rather than after.

Connect Claude Code and other clients

Bifrost speaks several provider dialects on one port, and OpenAI-compatible clients use http://10.0.1.50:8080/v1 with the virtual key as the API key and a provider/model name, which covers the OpenAI SDKs, curl and tools like Aider (see our Aider cheat sheet for its base URL flags). Anthropic clients use the /anthropic prefix, and Bifrost translates the Messages API onto whatever backend the model name points at:

curl -s localhost:8080/anthropic/v1/messages -H "Authorization: Bearer sk-bf-ci-bot-lab-0001" -H "anthropic-version: 2023-06-01" -H "Content-Type: application/json" \
  -d '{"model":"ollama/qwen2.5:0.5b","max_tokens":60,"messages":[{"role":"user","content":"Give one tip for writing a good git commit message."}]}' \
  | jq '{type, role, model, content: .content, stop_reason, usage}'

Ollama wrote the answer, yet it arrives in Anthropic’s shape with content blocks, a stop_reason and Anthropic-style usage:

{
  "type": "message",
  "role": "assistant",
  "model": "qwen2.5:0.5b",
  "content": [
    {
      "type": "text",
      "text": "When writing a good Git commit message, it's important to be concise yet descriptive. The message should clearly convey the changes made in the code, such as adding a new feature, fixing a bug, or changing a file. It should also be easy to understand for other developers and should include relevant information"
    }
  ],
  "stop_reason": "max_tokens",
  "usage": {
    "input_tokens": 16,
    "cache_creation_input_tokens": 0,
    "cache_read_input_tokens": 24,
    "cache_creation": {
      "ephemeral_5m_input_tokens": 0,
      "ephemeral_1h_input_tokens": 0
    },
    "output_tokens": 60
  }
}

Claude Code needs two environment variables: ANTHROPIC_BASE_URL pointing at the /anthropic prefix and ANTHROPIC_AUTH_TOKEN set to an sk-bf- virtual key. We ran Claude Code 2.1.317 in a throwaway Node container on the Compose network with a local model:

docker run --rm --network bifrost_default -e ANTHROPIC_BASE_URL=http://bifrost:8080/anthropic -e ANTHROPIC_AUTH_TOKEN=sk-bf-ci-bot-lab-0001 node:22-alpine \
  sh -c "npm i -g @anthropic-ai/claude-code >/dev/null 2>&1; claude --version; claude -p 'Reply with the single word ready' --model ollama/qwen2.5:0.5b"

The request went through the gateway, the logs tagged it as Claude Code, and the reply streamed back after 25 seconds. The reply itself was a ramble about a tool called “Ultracode”, most likely because Ollama’s default context window truncated Claude Code’s long system prompt (the logs show only 2,050 prompt tokens; we did not check num_ctx). Before you use a local model for agent work, raise the Ollama context length or serve it through vLLM with a larger --max-model-len, and pick a model trained for tool use. Our Claude Code install guide for Ubuntu covers the client side, and the Claude Code cheat sheet lists the settings file where the two variables can live permanently.

Manage the gateway from config.json

Learning Bifrost is quickest in the web UI, but a gateway you rebuild from Git is easier to trust. In the default mode config.json seeds the SQLite store at startup, each entity keeps a content hash, and an entity you change in the file overwrites its database copy on the next restart while UI-created entities are left alone. That’s why team-a, made in the UI, survived every restart next to ci-bot from the file. This is the exact config.json the lab ended with:

sudo vim ~/bifrost/data/config.json

It holds the Ollama provider, the flaky mock with retries, the ci-bot key with a 2,000 tokens an hour limit, the cache and the MCP server. The vLLM provider and team-a are absent because they live in the database from the UI steps:

{
  "$schema": "https://www.getbifrost.ai/schema",
  "client": {
    "enforce_auth_on_inference": true,
    "enable_logging": true
  },
  "providers": {
    "ollama": {
      "keys": [
        {
          "name": "ollama-local",
          "value": "",
          "models": [
            "*"
          ],
          "weight": 1.0,
          "ollama_key_config": {
            "url": "http://ollama:11434"
          }
        }
      ],
      "network_config": {
        "allow_private_network": true,
        "default_request_timeout_in_seconds": 120
      }
    },
    "flaky": {
      "keys": [
        {
          "name": "flaky-key",
          "value": "dummy",
          "models": [
            "*"
          ],
          "weight": 1.0
        }
      ],
      "network_config": {
        "base_url": "http://flaky:9000",
        "allow_private_network": true,
        "max_retries": 2,
        "retry_backoff_initial": 500,
        "retry_backoff_max": 2000
      },
      "custom_provider_config": {
        "base_provider_type": "openai",
        "allowed_requests": {
          "chat_completion": true,
          "chat_completion_stream": true
        }
      }
    }
  },
  "governance": {
    "virtual_keys": [
      {
        "id": "vk-ci-bot",
        "name": "ci-bot",
        "value": "sk-bf-ci-bot-lab-0001",
        "is_active": true,
        "provider_configs": [
          {
            "provider": "flaky",
            "allowed_models": [
              "*"
            ],
            "key_ids": [
              "*"
            ]
          },
          {
            "provider": "ollama",
            "allowed_models": [
              "qwen2.5:0.5b"
            ],
            "key_ids": [
              "*"
            ]
          }
        ],
        "rate_limit_id": "rl-ci-bot"
      }
    ],
    "rate_limits": [
      {
        "id": "rl-ci-bot",
        "token_max_limit": 2000,
        "token_reset_duration": "1h"
      }
    ]
  },
  "vector_store": {
    "enabled": true,
    "type": "redis",
    "config": {
      "addr": "redis:6379"
    }
  },
  "plugins": [
    {
      "enabled": true,
      "name": "semantic_cache",
      "config": {
        "provider": "ollama",
        "embedding_model": "nomic-embed-text",
        "dimension": 768,
        "threshold": 0.85,
        "ttl": "10m",
        "cache_by_model": true,
        "cache_by_provider": true,
        "default_cache_key": "lab"
      }
    }
  ],
  "mcp": {
    "client_configs": [
      {
        "name": "everything",
        "connection_type": "http",
        "connection_string": "http://everything:3001/mcp",
        "tools_to_execute": [
          "*"
        ]
      }
    ]
  }
}

The file is owned by UID 1000 inside the data directory, so edit it with sudo and apply it with docker compose restart bifrost. A real deployment would put the virtual key value in an environment variable with the env. prefix instead of in the file, and keep the file in Git without secrets. If you want the file to be the only source of truth, set config_store.enabled to false: the UI then stops accepting config edits and every change is a restart.

Fix the errors we hit

Every error below came from this lab, with the message as Bifrost printed it. Most of them are about the link between the gateway and something else, not about Bifrost being broken.

Error: “/app/data is not writable by UID:GID 1000:0 (owned by 1001:1001)”

Bifrost restarts in a loop and docker logs bifrost prints this line with hints for Kubernetes and OpenShift. The bind-mounted data directory belongs to your login user and the image runs as UID 1000, group 0. Run sudo chown -R 1000:0 ~/bifrost/data and restart. On Kubernetes, set fsGroup in the pod security context instead.

Error: “dashboard auth is not configured and no setup token is set”

Every /api call and /metrics returns 403 with this message, and the dashboard can’t create an admin account. Stop Bifrost, set BIFROST_SETUP_TOKEN (or setup_token in config.json) and start it again. With the token set but no admin yet, the API answers 401 and asks for the token in the X-Bifrost-Setup-Token header instead.

Error: “connection to private IP 172.19.0.5 is not allowed”

The response is a 502 of type provider_connection_failed, and the provider’s model row shows a red icon because the model list refresh failed the same way. Turn on Allow Private Network on the provider, or set allow_private_network: true in its network_config. Any backend on a LAN, a Docker network or a Kubernetes pod network needs it.

Error: “no keys found for provider: ollama and model: qwen2.5:0.5b”

We got this 400, type no_key_supports_model, on both providers right after updating a virtual key through the API, and it looked like a backend failure. It was the key: a PUT without key_ids set allow_all_keys to false. Read the key back with the API, then send the update again with "key_ids":["*"] on every provider config.

A budget never trips on a local model

Requests go through, current_usage stays at 0 and no 402 ever arrives. Bifrost has no price for self-hosted models, so their cost is zero. Create a pricing override for the model name, as shown in the budgets section, and the budget starts counting from the next request.

Chat replies come back empty after granting MCP tools

The response has finish_reason “stop”, empty content and a prompt token count far above the size of the message. The key’s MCP tools are injected into every chat request, and small models give up under the extra definitions. Send an empty x-bf-mcp-include-tools header from chat clients, or move the MCP grant to a separate key used only by agents.

So who is the free build for? It fits a small team running one box in front of a few local or hosted models, where virtual keys, budgets, failover and caching cover most of what a gateway needs to do, and a gateway that showed 158 MiB in docker stats at the end of our lab is easy to live with. A platform team that needs SSO, RBAC and audit logs around its per-team budgets is looking at the Enterprise tier: RBAC and audit logs are locked in the OSS image, and the docs list SSO as Enterprise. Language matters less than you’d expect: clients only see an OpenAI or Anthropic compatible endpoint, though a Go shop also gets the option of embedding Bifrost as a library instead of running it as a separate service.

Ollama behaves the same behind any gateway. Our Ollama install guide for Ubuntu 26.04 covers that backend in more depth, and the coding agent comparison helps you decide which clients will sit in front of the gateway.

Keep reading

Ollama Models Cheat Sheet: Best Models by VRAM, Tested (PDF) AI Ollama Models Cheat Sheet: Best Models by VRAM, Tested (PDF) Open Source LLM Comparison Table (2026) AI Open Source LLM Comparison Table (2026) Claude Code Cheat Sheet: Commands, Shortcuts and Flags (PDF) AI Claude Code Cheat Sheet: Commands, Shortcuts and Flags (PDF) Dual GPU LLM Build: PCIe x8/x8 vs x16/x4, PSU and Cooling AI Dual GPU LLM Build: PCIe x8/x8 vs x16/x4, PSU and Cooling OpenCloud vs ownCloud vs Nextcloud: Tested on Identical VMs Cloud OpenCloud vs ownCloud vs Nextcloud: Tested on Identical VMs How To Manage Kubernetes Cluster with Kubevious Containers How To Manage Kubernetes Cluster with Kubevious

Leave a Comment

Press ESC to close