On the same three CPU cores, Bifrost served up to 1,540 requests per second with logging on, and a tuned LiteLLM proxy topped out near 110. That gap is real, but it is not the whole LiteLLM vs Bifrost story. LiteLLM ships the MCP gateway that the vendor comparison pages say it lacks, and both gateways survived a killed backend without a single failed request.
The throughput numbers come from a re-run of the mock-server benchmark that Bifrost publishes, on two Proxmox VMs. Each gateway was pinned to cores 0-2, and LiteLLM got workers, Postgres and Redis instead of the single-worker default. At 100 requests per second Bifrost added 2.4 ms at the median and LiteLLM 30 to 55 ms (LiteLLM v1.106.6 and Bifrost v2.4.0, benchmarked November 2026 on Ubuntu 24.04). The streaming test ran through Ollama on CPU because no GPU was available for vLLM. The enterprise rows of the matrix come from vendor docs and dashboards, not from a licensed install.
LiteLLM vs Bifrost results at a glance
Bifrost wins every throughput and footprint row by a wide margin, and the two are level on failover, MCP and streaming within the noise of a CPU model. The table holds the measured values. Each row links back to the section that explains how it was taken.
| Measured on the test box | Bifrost (logging on) | LiteLLM (3 workers, Postgres, Redis) |
|---|---|---|
| Added p50 latency at 100 req/s, vs the mock alone | +2.4 ms | +30.5 to +54.9 ms (two passes) |
| Highest offered rate with 100% success before queueing | 1,000 req/s | 100 req/s |
| Peak served throughput, logging on / off | 1,540 / 1,995 req/s | 110 / 161 req/s |
| Streaming TTFT p50 through Ollama (direct: 254 ms) | 274 ms | 290 ms |
| Key that always gets HTTP 429, 3,000 requests | 100% success, p99 661 ms | 100% success, p99 1,283 ms |
| Backend killed 10 s into a 30 s run | 100% success, p99 66 ms | 100% success, p99 179 ms |
| MCP gateway in the open source image | Yes, tested | Yes, tested |
| Fresh start to healthy | 12.7 to 20.9 s | 44.7 to 112.8 s |
| Idle memory | 319 MB (128 MB without log store) | 2.1 GB plus 54 MB Postgres |
| Image size on disk | 412 MB | 1.72 GB |
| Licence of the gateway code | Apache-2.0 | MIT, except the proprietary enterprise/ directory |
The throughput rows decide the choice only when the gateway itself is the bottleneck, which is rarer than the vendor charts suggest. A real model at 100 req/s needs a GPU cluster. On our CPU model the gap shrank to 16 ms of TTFT, roughly 6% of the first-token time. The picks below follow from that:
- A small team on one box: Bifrost. One static binary in a 412 MB image, 128 to 319 MB at idle and no database to run, so it fits beside the model server.
- A platform team that needs per-team budgets and SSO without a contract: LiteLLM. Teams, budgets, guardrails and the admin UI all load in the open source image, and SSO is free for up to 5 users.
- Thousands of requests per second through one gateway: Bifrost, or LiteLLM with far more cores. On this CPU one LiteLLM worker handled 35 to 55 req/s, so 1,000 req/s means roughly 20 to 30 workers.
- A Python shop that extends the gateway: LiteLLM. Its callbacks and guardrail hooks are Python, and tools such as Aider already use the same library underneath.
Bifrost gates clustering, guardrails, RBAC, audit logs and user management behind its enterprise licence, and the dashboard says so on each page. LiteLLM gates organizations, audit logs and SSO past 5 users. Neither split is hidden, but they fall in different places, so check the matrix further down against the features you need before you trust the throughput row.
How the test rig was built
The rig follows the method in maximhq/bifrost-benchmarking: a Go mock provider with a fixed 60 ms latency stands in for OpenAI, and Vegeta fires a constant request rate at the gateway. Using the vendor tool removes the argument that the harness favours one side. The difference is in the gateway setup, which the vendor repo does not publish for LiteLLM.

Both VMs ran on one Proxmox node with an Intel Core i7-6700 (4 cores, 8 threads) that was otherwise below 5% CPU. Only one gateway ran at a time. How each number was taken:
- Pinning: the gateway container got
--cpuset-cpus 0-2on bench01, and the LiteLLM Postgres and Redis containers got core 3. On bench02 the mock ran undertaskset -c 0,1and Vegeta undertaskset -c 2,3. - Latency and throughput: the Vegeta JSON report for a 30 s constant-rate run with a 30 s timeout, 10 s apart. Overhead is the gateway p50 minus the p50 of the same ladder fired straight at the mock.
- Auth on both: every request carried a virtual key, because Bifrost enforces one by default and LiteLLM refuses to start without a master key.
- Memory and CPU:
docker statssampled every 3 to 5 s during each ladder. Idle memory is the reading 20 s after the health check passed. - Start time: wall clock from
docker runordocker compose upuntil/healthor/health/livelinessanswered.
LiteLLM got what its production best practices page asks for, on the litellm-database image: one worker per vCPU (3), Postgres for keys and spend, Redis for the transaction buffer, proxy_batch_write_at: 60, a pool limit of 10 and error logs off. Two caveats on that tuning. The page sets workers equal to vCPUs only for a single VM (on Kubernetes it runs one worker per pod and scales pods), and it documents the Redis transaction buffer for roughly 1,000 req/s or ten-plus instances, far above this rig, but we enabled it anyway. The multi-instance setup it describes was not tested, because the lab allowed three cores per gateway, so any figure above one instance in this article is a projection. A default single-worker run is in the results too, because that is the setup the vendor comparisons appear to use. The images that ran, with their digests:
maximhq/bifrost:v2.4.0 sha256:a1c90607dcfd13c1fb2f9476ceb3bb24407a50ade7fdf3e61a59fb42b09e5c57 412MB
ghcr.io/berriai/litellm:v1.106.6 sha256:37e2dafb088a7723beb998590682957386dfc83ffea1848486f8e0e5cfac0b06 1.81GB
ghcr.io/berriai/litellm-database:v1.106.6 sha256:7ef0e78fe00ad5ea7ea45666cc1d1f02cb7076aee69091543bf397de52db33ce 1.72GB
redis:7-alpine sha256:8da19c7399dca900c6822a1cf30f9d2285585e2c4b457ff4095b0f4500054d3f 57.8MB
postgres:16-alpine sha256:0cefaabd882e42bcb2a2f3dc56d3ce4770560ba62fd7f197f1a2e008fd5731ab 420MB
Four things broke while the rig was being built, and each one cost time that a reader setting up either gateway will also spend. They are listed here with the exact text, because the messages are what people search for.
Bifrost: “connection to private IP 192.168.1.118 is not allowed”
Bifrost blocks provider URLs on private networks by default, so a mock, an Ollama box or a vLLM server on the LAN returns HTTP 502 with provider_connection_failed. The model list fails the same way at start, which surfaces first as the less helpful no keys found for provider. The fix is "allow_private_network": true in the provider network_config. A second trap sits next to it: the built-in openai provider ignores base_url, so a self-hosted OpenAI-compatible backend needs a custom provider with base_provider_type: openai.
LiteLLM: “Fatal glibc error: CPU does not support x86-64-v2”
The LiteLLM image exited at once on a Proxmox VM using the default kvm64 CPU type, while the Bifrost container on the same VM ran fine. Setting the VM CPU type to host and restarting it fixed it. Any hypervisor that exposes a baseline x86-64 CPU model will hit this.
LiteLLM: “use_redis_transaction_buffer is enabled in general_settings but no Redis is configured”
The production page recommends use_redis_transaction_buffer, and router_settings.redis_host does not satisfy it. The proxy refused to start until REDIS_HOST and REDIS_PORT were set as environment variables. The error offers two fixes, a Redis response cache or the REDIS_* variables. We took the variables, because a cache would answer identical benchmark requests without calling the mock.
LiteLLM: “proxy refused to start: no master key is set”
This LiteLLM release will not boot without a master key unless you set LITELLM_DANGEROUSLY_PERMIT_WEAK_OR_UNSET_MASTER_KEY=true. So the open, keyless proxy that older comparisons benchmarked is no longer the default, and every LiteLLM run here used the master key or a generated virtual key.
Step 1: Measure gateway overhead against a 60 ms mock
Straight at the mock, Vegeta held a p50 between 60.6 and 60.9 ms at every rate from 100 to 5,000 req/s. The one blemish was 127 failed requests at 3,000 req/s, because localhost resolved to ::1 and the mocker listens on IPv4 only. Bifrost added 2.2 to 3.9 ms at the median up to 1,000 req/s, with all 30,000 requests at 1,000 req/s returning 200:

Rows above 1,000 req/s in that capture say more about the load box than about Bifrost. Status 0 is the Vegeta code for any transport error, and here the errors were bind: address already in use on bench02: the client ran out of ephemeral ports. It sent only about 1,050 to 1,130 req/s of the 3,000 and 5,000 asked for. After widening the port range on the load box (Step 6), the same config held 1,500 req/s with every request successful and a p50 of 152 ms. At 2,000 and 3,000 offered it served 1,499 to 1,540 req/s while requests queued in its buffer. Bifrost did not fail requests past its ceiling, it queued them. The tuned LiteLLM stack hit the same wall at about one fourteenth of that rate:

At 100 req/s every LiteLLM request succeeded with a p50 of 115.8 ms, which is 55 ms of gateway time on top of the 60 ms mock. At 250 req/s success fell to 80%, with 144 HTTP 503 responses and 1,303 status 0 errors. The more useful finding is what happens past saturation: served throughput dropped from 110 req/s at 500 offered to 30 to 48 req/s at 1,000 and above, with connection reset by peer and client timeouts as the errors. Bifrost kept serving its ceiling under overload, while LiteLLM collapsed, so a LiteLLM deployment needs its rate limit set below its measured ceiling rather than at it.
Separating out logging changes both numbers. With disable_spend_logs: true, the LiteLLM ceiling rose from about 110 to 161 req/s. Its p50 at 100 req/s also fell from 115.8 to 91.4 ms in that pass, but a later run with spend logs on measured the same 91.4 ms (its p99 was 832 ms, against 318 ms for the first spend-logs-on run). The median gap is run-to-run drift. The ceiling is the real cost of spend logging. With the log store and request logging off, the Bifrost p50 at 1,000 req/s barely moved (64.6 to 63.1 ms), but its ceiling rose from about 1,540 to 1,995 req/s. Logging is not free in either product. In Bifrost the default SQLite log store is what caps throughput first:
| Configuration | Offered req/s | Success | Served req/s | p50 | p99 |
|---|---|---|---|---|---|
| Mock only (baseline) | 1,000 | 100% | 997 | 60.6 ms | 62.2 ms |
| Bifrost, logging on | 1,000 | 100% | 997 | 64.6 ms | 175.6 ms |
| Bifrost, logging on, tuned client | 1,500 | 100% | 1,490 | 152.4 ms | 336.4 ms |
| Bifrost, logging off, tuned client | 2,000 | 100% | 1,995 | 73.7 ms | 518.2 ms |
| LiteLLM tuned, spend logs on (first pass) | 100 | 100% | 99 | 115.8 ms | 318.0 ms |
| LiteLLM tuned, spend logs off | 100 | 100% | 99 | 91.4 ms | 144.5 ms |
| LiteLLM tuned, spend logs off | 150 | 100% | 146 | 880.4 ms | 2,279 ms |
| LiteLLM default, 1 worker, no DB | 50 | 100% | 49 | 2,381 ms | 6,850 ms |
The last row is the setup most published comparisons describe. One LiteLLM worker already queued requests at 25 req/s (p95 7.6 s), which is why the vendor charts show multi-second LiteLLM latencies. Three workers on three cores lifted the clean ceiling to about 100 req/s and the peak from 49 to 110 req/s, so the tuning roughly doubled throughput and the per-core rate stayed near 35 to 55 req/s. CPU, not configuration, is what LiteLLM runs out of: docker stats showed its three pinned cores saturated in every overloaded step, and the Bifrost cores were saturated the same way above 1,500 req/s.
Step 2: Measure streaming TTFT through a real model
A mock proves overhead, but users feel time to first token. The streaming test sent 20 sequential requests (after one warm-up) for qwen2.5:0.5b on Ollama, running on the four CPU cores of bench02, and timed the first SSE chunk that carried content. We would have run this on vLLM, as in the vLLM production install, but the GPU budget for this run was zero. Ollama on CPU stood in as the real backend. The setup matches the Ollama install guide, in a container.
direct runs=20 ttft_p50=253.9ms ttft_min=234.0ms ttft_max=314.6ms total_p50=2159.3ms
bifrost runs=20 ttft_p50=273.7ms ttft_min=240.2ms ttft_max=353.3ms total_p50=2237.0ms
litellm runs=20 ttft_p50=289.8ms ttft_min=245.9ms ttft_max=781.5ms total_p50=2156.1ms
direct-rerun runs=20 ttft_p50=246.1ms ttft_min=219.5ms ttft_max=294.5ms total_p50=2184.5ms
Bifrost added about 20 to 28 ms to TTFT and LiteLLM about 36 to 44 ms, depending on which direct run you take as the baseline. That is more than the Bifrost overhead on the mock, because both gateways went through their own Ollama provider (ollama_chat/ in LiteLLM, ollama in Bifrost) while the direct run used the Ollama OpenAI endpoint. The format translation is part of the number. The total time to finish 64 tokens stayed within 81 ms across all four runs, inside the run-to-run noise of a CPU model. On a GPU serving tens of tokens per second the gap would matter more for TTFT and less for total time. The Ollama vs vLLM vs llama.cpp results show how much the backend itself moves those numbers.
Sequential requests flatter every gateway, so a second pass ran 8, 16 and 32 simultaneous streams (3 rounds each, 32 tokens per reply) with Ollama set to OLLAMA_NUM_PARALLEL=8. This pass used fresh clones of the same template with the same image digests, and each concurrency level ran direct, through Bifrost and through LiteLLM back to back:
direct streams=8 n=24 errors=0 ttft_p50=1419ms ttft_p95=1780ms total_p50=5359ms total_p95=5387ms
bifrost streams=8 n=24 errors=0 ttft_p50=1372ms ttft_p95=1534ms total_p50=5344ms total_p95=5350ms
litellm streams=8 n=24 errors=0 ttft_p50=1546ms ttft_p95=1764ms total_p50=5072ms total_p95=5369ms
direct streams=16 n=48 errors=0 ttft_p50=3380ms ttft_p95=6695ms total_p50=7172ms total_p95=10046ms
bifrost streams=16 n=48 errors=0 ttft_p50=3940ms ttft_p95=6861ms total_p50=7817ms total_p95=10932ms
litellm streams=16 n=48 errors=0 ttft_p50=3742ms ttft_p95=7312ms total_p50=8218ms total_p95=11140ms
direct streams=32 n=96 errors=0 ttft_p50=8914ms ttft_p95=16092ms total_p50=12995ms total_p95=20362ms
bifrost streams=32 n=96 errors=0 ttft_p50=8830ms ttft_p95=17321ms total_p50=13277ms total_p95=21996ms
The LiteLLM row at 32 streams is missing because the Proxmox node rebooted in the middle of it, with all eight host threads busy. An earlier pass of the same test, 2 rounds per level, had completed it: LiteLLM 11.6 s TTFT p50 at 32 streams, against 9.8 s direct and 10.4 s through Bifrost in that pass. Read together, the two passes say the backend queue dominates once streams outnumber the model’s parallel slots.
At 8 and 16 streams the gateways landed within 560 ms of direct in either direction, which is inside the run-to-run spread (direct alone moved from 3.9 to 3.4 s at 16 streams between passes). Only at 32 streams did LiteLLM fall behind, by about 1.8 s at the median in the one pass that finished it, which fits a Python gateway with three workers juggling 32 open SSE streams. Treat that single point as a lead to re-test on a GPU backend, not as a result.
Step 3: Break a key and kill a backend
Both gateways passed both failure tests with zero failed requests at 100 req/s, the rate LiteLLM could sustain. The mock was restarted with -rate-limited-keys "Bearer mock-key-429", so one of two upstream keys answered every call with HTTP 429. For the second test, a second mock on port 8001 sat behind the same model name and was killed 10 s into a 30 s run:

LiteLLM handled both through its router: two deployments under one model_name, num_retries: 2, allowed_fails: 1 and a 60 s cooldown. Its p99 on the 429 test was 1,283 ms because a request that drew the bad key paid for a retry. Bifrost needed no retry settings beyond max_retries on the 429 provider. On the kill test its virtual key split traffic 50/50 across two providers, and after the kill every request was ultimately served by the survivor, with a p99 of only 65.5 ms.
Bifrost log counters explain that low p99, and they also show a weakness. After both runs Bifrost reported 6,978 upstream attempts for 6,000 client requests, an attempt success rate of 85.98% against a user-facing success rate of 100%. The provider behind the killed mock logged 1,467 attempts, close to half of the 3,000 requests, so Bifrost never stopped choosing it. For the last 20 s about every second request tried the dead backend first and was retried on the survivor, which logged 2,511.
That cost almost nothing here because a closed port refuses at once. We did not test a backend that hangs instead, but on this evidence it would add the provider timeout to about every second request, and the Bifrost circuit breaker is an enterprise feature that its docs describe as triggered by response header signals. In LiteLLM, allowed_fails and cooldown_time put a failing deployment on cooldown in the open source build, although a hung call still waits for request_timeout (600 s in this config) before it counts as a failure, so set that timeout low on a router you rely on for failover.
Step 4: Test the MCP gateway claim
The Maxim feature-by-feature comparison says “LiteLLM does not provide a native MCP gateway”. That is wrong for the open source LiteLLM image we ran. A small FastMCP server with two tools was added to both gateway configs. A real MCP client (the Python mcp SDK over streamable HTTP) connected to the /mcp endpoint of each gateway, listed the tools and called one:

Both gateways prefix tool names with the server name (lab_tools-add), and both deny tools to keys by default, which is the right default. A fresh LiteLLM virtual key got HTTP 403 with “The key has no MCP servers granted” until it was created with object_permission.mcp_servers. The Bifrost virtual key listed zero tools until its mcp_configs named the client. Per-key tool scoping therefore works in both open source images. Virtual MCPs in Bifrost, which bundle tools from several servers into one endpoint, are an enterprise feature according to its docs, and we did not test them.
Step 5: Compare cold start, image size and memory
Bifrost reached a healthy /health in a median of 15.6 s across ten fresh starts (12.7 to 20.9 s), and its own log put bootstrap at 12.6 s. That is slow for a Go binary, and the start log is mostly config-store and log-store migrations. LiteLLM took 112.8 s on its first start against an empty Postgres and 44.7 to 89.0 s on later fresh stacks. It needed 35.8 s to restart the proxy alone against an existing database. Without a database, the single-worker image took 70.7 s, so most of the LiteLLM start time is not the database.
| Footprint | Bifrost | LiteLLM |
|---|---|---|
| Image on disk (compressed) | 412 MB (104 MB) | 1.72 GB (403 MB), database image |
| Idle memory | 128 MB without log store, 319 MB with it | 541 MB with 1 worker, 2.1 GB with 3 workers |
| Peak memory during the ladder | 4.0 GiB without log store, 6.0 GiB with it | 2.7 GiB proxy, 302 MiB Postgres, 6 MiB Redis |
| Average CPU at 100 req/s | 27% of one core | 202% (2.0 cores), plus 4% Postgres and 3% Redis |
| Average CPU at 1,000 req/s | 213% (2.1 cores) | Not reachable (ceiling about 110 req/s) |
| Extra services required | None (SQLite inside the volume) | Postgres for keys and spend, Redis for multi-worker |
The CPU rows average the docker stats samples (one every 3 to 5 s) taken while a dedicated 30 s run was under load, in the second lab pass with the spend-logs-on LiteLLM config. That LiteLLM run measured a 91.4 ms p50 at 100 req/s, 24 ms better than the first pass, which is the size of the run-to-run drift on this stack. The CPU rows explain the throughput gap better than any latency figure: Bifrost at 1,000 req/s used about the same CPU as LiteLLM at 100 req/s, so per request LiteLLM spends 7.5 to 9.5 times the compute. The peak memory row is the one that would worry us in production. Bifrost grew to 6.0 GiB while it queued requests above its ceiling, with up to about 20,000 client connections open, a 10,000-deep provider buffer and drop_excess_requests: false in the config. On a host with less RAM than that, overload would end in the OOM killer rather than in slow responses. We did not re-run with dropping enabled, but we would set drop_excess_requests: true and size buffer_size to the RAM available. LiteLLM stayed within 0.7 GB of its idle memory under overload because it rejected and timed out requests instead.
Open source vs enterprise feature matrix
The matrix marks where each claim comes from, because only part of it could be tested without a licence. “Lab” means it ran in this benchmark. “UI” means the page opened in the open source dashboard, or showed a licence notice. “Docs” means it is taken from the vendor enterprise or feature pages and was not run:
| Feature | Bifrost open source | LiteLLM open source |
|---|---|---|
| Virtual keys | Yes (lab) | Yes (lab) |
| Teams | Page opens (UI) | Yes (UI) |
| Budgets and rate limits | Yes (docs) | Yes (UI, docs) |
| Fallbacks and load balancing | Yes (lab) | Yes (lab) |
| Exact response caching | Yes (UI) | Yes, Redis (UI) |
| Semantic caching | Yes (docs) | Yes, redis-semantic, qdrant-semantic or valkey-semantic (docs) |
| Guardrails | Enterprise (UI) | Presidio and custom guardrails open, several integrations enterprise (docs) |
| MCP gateway with per-key tools | Yes (lab); Virtual MCPs enterprise (docs) | Yes (lab) |
| SSO | Enterprise, OIDC and SCIM (UI, docs) | Free up to 5 users, licence beyond (docs) |
| Prometheus and OpenTelemetry | Yes (UI) | Yes (docs) |
| Request logs and usage UI | Yes (lab) | Yes (lab) |
| RBAC, users, audit logs | Enterprise (UI) | Audit logs and RBAC enterprise (docs) |
| Organizations, business units | Enterprise (UI) | Enterprise (UI) |
| Clustering and adaptive routing | Enterprise (UI); multinode via shared config (docs) | Multiple instances on shared Postgres and Redis (docs) |
The docs-sourced rows rest on these vendor pages, read in November 2026:
- LiteLLM open source vs Enterprise: the enterprise page lists SSO (free up to 5 users), audit logs, RBAC, organizations and the paid guardrail integrations.
- LiteLLM budgets, caching and metrics: budgets and rate limits, caching (including the semantic cache types), guardrails and Prometheus.
- Bifrost open source vs Enterprise: the enterprise overview lists clustering, adaptive routing, the circuit breaker, Virtual MCPs, OIDC and SCIM, RBAC, audit logs and guardrails.
- Bifrost governance and caching: budgets and limits, semantic caching, MCP tool filtering and multinode without the enterprise build.
The licences follow the same split. The Bifrost repository is Apache-2.0 and its enterprise build ships as a separate private image. The LiteLLM root LICENSE is MIT except for the enterprise/ directory, which needs a subscription for production use, and that is why GitHub reports the licence as NOASSERTION. The practical difference is the governance layer: LiteLLM gives away more of it in the open image, while Bifrost gives away more raw performance.
The two dashboards side by side
Both gateways ship a web UI in the same container, and both logged the six Ollama requests sent to them after the benchmark. The Bifrost LLM Logs page puts request count, success rate, average latency and token totals above the log table:

Bifrost v2.4.0 also locks its management API until you either create an admin account or send the X-Bifrost-Setup-Token header, and the dashboard keeps a warning in the sidebar until you do. The LiteLLM request log shows duration and a TTFT column per call, which is useful for spotting a slow backend without leaving the UI:

In the red banner, LiteLLM warns that anyone holding the master key can log in as admin. It asks you to create a real admin account and then set disable_env_credential_login: true, and it is worth doing before the UI is reachable from anywhere. The MCP server defined in config.yaml showed up as healthy on the MCP Servers page:

Pages like these are where the enterprise boundaries in the matrix come from. The open source Bifrost dashboard lists Cluster, Guardrails, Adaptive Routing and the user management pages in the sidebar, then shows a licence notice when you open them:

Of the twelve LiteLLM pages we opened, only Organizations carried a licence notice. Logs, MCP servers, guardrails, policies, teams, budgets, usage, caching, access groups, internal users and Lens all loaded without a key:

A page that loads is not proof that every option on it works without a licence, which is why the matrix marks those rows as UI rather than lab.
Step 6: Reproduce the benchmark
You need two Linux VMs with 4 vCPUs each, Docker and a CPU type that exposes x86-64-v2. Replace 192.168.1.150 (the gateway box) and 192.168.1.118 (the load box) with your own addresses in the files below. Every file and script below is the one the numbers came from, printed in full because the harness is not in a public repository yet. Install the tools on both VMs:
sudo apt-get update -qq
sudo apt-get install -y -qq docker.io docker-compose-v2 jq sysstat bc git python3-venv
sudo usermod -aG docker $USER
sudo systemctl disable --now unattended-upgrades apt-daily.timer apt-daily-upgrade.timer
Disabling the upgrade timers matters more than it looks. On our first pass, apt upgraded packages on both boxes in the middle of a ladder and restarted sshd, and that run had to be thrown away. On the load box, also widen the ephemeral port range, because Vegeta opens a new connection for every request it cannot reuse and ran out of ports above about 1,000 req/s. The setting does not survive a reboot, and even with it the client still hit a few port errors at 5,000 req/s offered:
sudo sysctl -w net.ipv4.ip_local_port_range="1024 65535" net.ipv4.tcp_tw_reuse=1
On the load box, build the mocker, the Bifrost benchmark tool and the Vegeta CLI from source inside a Go container, so nothing but Docker has to be installed on the host:
git clone -q https://github.com/maximhq/bifrost-benchmarking.git ~/bifrost-benchmarking && cd ~/bifrost-benchmarking
# https://github.com/tsenart/vegeta/releases/tag/v12.12.0
sudo docker run --rm -v $PWD:/src -v $HOME/bin:/out -w /src -e CGO_ENABLED=0 -e GOFLAGS=-buildvcs=false golang:1.24 sh -c "cd mocker && go build -o /out/mocker . && cd .. && go build -o /out/benchmark benchmark.go && GOBIN=/out go install github.com/tsenart/vegeta/[email protected]"
Start the mock on cores 0 and 1 with a fixed 60 ms latency. The -host 0.0.0.0 flag matters, because the mocker binds to localhost by default and the gateway box would get connection refused:
nohup taskset -c 0,1 ~/bin/mocker -host 0.0.0.0 -port 8000 -latency 60 > ~/mocker.log 2>&1 < /dev/null &
On the gateway box, create the Bifrost config. The custom mock provider, allow_private_network and the virtual key declared in the file are the three parts that make it work against a LAN backend:
mkdir -p ~/harness/bifrost ~/harness/litellm
vim ~/harness/bifrost/config.json
Paste the configuration:
{
"$schema": "https://www.getbifrost.ai/schema",
"client": {
"initial_pool_size": 5000,
"drop_excess_requests": false,
"enable_logging": true
},
"providers": {
"mock": {
"keys": [
{ "name": "mock-a", "value": "mock-key-a", "models": ["gpt-4o-mini"], "weight": 1.0 }
],
"network_config": {
"base_url": "http://192.168.1.118:8000",
"default_request_timeout_in_seconds": 60,
"allow_private_network": true
},
"concurrency_and_buffer_size": {
"concurrency": 5000,
"buffer_size": 10000
},
"custom_provider_config": {
"base_provider_type": "openai",
"allowed_requests": { "chat_completion": true, "chat_completion_stream": true }
}
}
},
"governance": {
"virtual_keys": [
{
"id": "vk-bench",
"name": "bench",
"value": "sk-bf-bench-0001",
"is_active": true,
"provider_configs": [
{ "provider": "mock", "weight": 1.0, "allowed_models": ["*"], "key_ids": ["*"] }
]
}
]
},
"config_store": {
"enabled": true,
"type": "sqlite",
"config": { "path": "/app/data/config.db" }
},
"logs_store": {
"enabled": true,
"type": "sqlite",
"config": { "path": "/app/data/logs.db" }
}
}
For the logging-off runs, the same file has enable_logging and logs_store.enabled set to false. The data directory is wiped on every start so an old config.db cannot carry settings over. Start Bifrost pinned to cores 0-2 on the host network, which is how both gateways ran so neither paid for Docker’s NAT:
sudo rm -rf ~/bf-data; mkdir -p ~/bf-data && cp ~/harness/bifrost/config.json ~/bf-data/config.json
# https://github.com/maximhq/bifrost/releases/tag/transports/v2.4.0
sudo docker run -d --name bifrost --cpuset-cpus 0-2 --network host -e BIFROST_SETUP_TOKEN=lab-setup-token -v ~/bf-data:/app/data maximhq/bifrost:v2.4.0
until curl -sf localhost:8080/health >/dev/null; do sleep 0.2; done
LiteLLM needs a config and a Compose file. The config carries the production settings from the rig section:
vim ~/harness/litellm/config.yaml
Add the model, router and general settings:
model_list:
- model_name: mock/gpt-4o-mini
litellm_params:
model: openai/gpt-4o-mini
api_base: http://192.168.1.118:8000/v1
api_key: mock-key-a
litellm_settings:
set_verbose: false
json_logs: true
request_timeout: 600
router_settings:
routing_strategy: simple-shuffle
redis_host: 127.0.0.1
redis_port: 6379
general_settings:
master_key: os.environ/LITELLM_MASTER_KEY
database_url: os.environ/DATABASE_URL
proxy_batch_write_at: 60
database_connection_pool_limit: 10
disable_error_logs: true
use_redis_transaction_buffer: true
The spend-logs-off variant adds disable_spend_logs: true under general_settings. Now the Compose file, which pins the proxy to cores 0-2 and the two data services to core 3:
vim ~/harness/litellm/compose.yaml
Paste the three services:
services:
postgres:
image: postgres:16-alpine
cpuset: "3"
network_mode: host
environment:
POSTGRES_USER: litellm
POSTGRES_PASSWORD: litellm-lab-pass
POSTGRES_DB: litellm
healthcheck:
test: ["CMD-SHELL", "pg_isready -U litellm"]
interval: 2s
retries: 30
redis:
image: redis:7-alpine
cpuset: "3"
network_mode: host
litellm:
image: ghcr.io/berriai/litellm-database:v1.106.6
cpuset: "0-2"
network_mode: host
depends_on:
postgres:
condition: service_healthy
redis:
condition: service_started
environment:
LITELLM_MASTER_KEY: sk-lab-master-0001
LITELLM_SALT_KEY: sk-lab-salt-0001
DATABASE_URL: postgresql://litellm:[email protected]:5432/litellm
REDIS_HOST: 127.0.0.1
REDIS_PORT: "6379"
LITELLM_LOG: ERROR
LITELLM_MODE: PRODUCTION
volumes:
- ./${LITELLM_CONFIG:-config.yaml}:/app/config.yaml:ro
command: ["--config", "/app/config.yaml", "--port", "4000", "--num_workers", "3"]
Stop Bifrost first so only one gateway holds the cores, then bring the stack up and generate a virtual key for the load test. The first start against an empty database is the slow one, so the loop waits up to five minutes:
sudo docker rm -f bifrost
cd ~/harness/litellm && sudo docker compose up -d
for i in $(seq 1 600); do curl -sf localhost:4000/health/liveliness >/dev/null && break; sleep 0.5; done
KEY=$(curl -s -X POST localhost:4000/key/generate -H "Authorization: Bearer sk-lab-master-0001" -H "Content-Type: application/json" -d '{"key_alias":"bench"}' | jq -r .key); echo "$KEY"
The ladder script runs on the load box. It writes one Vegeta JSON report per rate into ~/results and prints a summary line, and it pins Vegeta to cores 2 and 3 so it never competes with the mock:
vim ~/ladder.sh
Add the script:
#!/bin/bash
# Run on the load-generator box. usage: ladder.sh <label> <url> <auth-header> <model> <rates...>
# Writes results/<label>-<rate>.json (vegeta report) and prints one summary line per rate.
set -u
label=$1; url=$2; auth=$3; model=$4; shift 4
mkdir -p ~/results
body=$(printf '{"model":"%s","messages":[{"role":"user","content":"Summarise the benefits of an LLM gateway in one line."}]}' "$model" | base64 -w0)
for rate in "$@"; do
printf '{"method":"POST","url":"%s","body":"%s","header":{"Content-Type":["application/json"],"Authorization":["%s"]}}\n' \
"$url" "$body" "$auth" > /tmp/target.json
taskset -c 2,3 ~/bin/vegeta attack -format=json -targets=/tmp/target.json -rate="${rate}/s" -duration=30s -timeout=30s \
| ~/bin/vegeta report -type=json > ~/results/"$label-$rate".json
jq -r --arg l "$label" --arg r "$rate" '"\($l) rate=\($r) ok=\((.success*100)|floor)% thr=\(.throughput|floor) p50=\(.latencies["50th"]/1e6|.*100|floor/100)ms p95=\(.latencies["95th"]/1e6|.*100|floor/100)ms p99=\(.latencies["99th"]/1e6|.*100|floor/100)ms codes=\(.status_codes|tostring)"' ~/results/"$label-$rate".json
sleep 10
done
Make it executable and run the baseline straight at the mock first. If the mock alone cannot hold a flat p50 at your top rate, the load box is the bottleneck and the gateway numbers mean nothing:
chmod +x ~/ladder.sh
~/ladder.sh direct http://localhost:8000/v1/chat/completions 'Bearer mock-key-a' gpt-4o-mini 100 250 500 1000 2000 3000 5000
Then run the same rates through each gateway. On the load box, set KEY to the LiteLLM key the gateway box printed; the Bifrost key is the one declared in its config:
~/ladder.sh bifrost http://192.168.1.150:8080/v1/chat/completions 'Bearer sk-bf-bench-0001' mock/gpt-4o-mini 100 250 500 1000 2000 3000 5000
~/ladder.sh litellm-tuned http://192.168.1.150:4000/v1/chat/completions "Bearer $KEY" mock/gpt-4o-mini 100 250 500 1000 2000 3000 5000
The streaming tests in Step 2 use two small Python scripts that need nothing beyond the standard library. The first times sequential streams:
vim ~/ttft.py
Paste the sequential TTFT script.
#!/usr/bin/env python3
"""Streaming TTFT through an OpenAI-compatible endpoint.
usage: ttft.py <url> <auth-header> <model> <runs>"""
import json, statistics, sys, time, urllib.request
url, auth, model, runs = sys.argv[1], sys.argv[2], sys.argv[3], int(sys.argv[4])
ttfts, totals = [], []
for i in range(runs + 1):
body = json.dumps({
"model": model, "stream": True, "max_tokens": 64, "temperature": 0,
"messages": [{"role": "user", "content": f"Run {i}: list three uses of an LLM gateway."}],
}).encode()
req = urllib.request.Request(url, data=body, method="POST",
headers={"Content-Type": "application/json", "Authorization": auth})
t0 = time.perf_counter(); first = None
with urllib.request.urlopen(req, timeout=120) as r:
for raw in r:
line = raw.decode().strip()
if not line.startswith("data:") or line == "data: [DONE]":
continue
chunk = json.loads(line[5:])
delta = (chunk.get("choices") or [{}])[0].get("delta", {}).get("content")
if delta and first is None:
first = time.perf_counter() - t0
total = time.perf_counter() - t0
if i == 0:
continue # warm-up run, model load
ttfts.append(first * 1000); totals.append(total * 1000)
print(f"runs={runs} ttft_p50={statistics.median(ttfts):.1f}ms ttft_min={min(ttfts):.1f}ms "
f"ttft_max={max(ttfts):.1f}ms total_p50={statistics.median(totals):.1f}ms")
Both Step 2 tests need Ollama and the gateways’ Ollama configs, which the concurrent block further down starts and describes, so set those up first. The sequential numbers came from 20 runs per target, with the label in front of each line added by sed:
python3 ~/ttft.py http://localhost:11434/v1/chat/completions "Bearer x" qwen2.5:0.5b 20 | sed 's/^/direct /'
python3 ~/ttft.py http://192.168.1.150:8080/v1/chat/completions "Bearer sk-bf-bench-0001" ollama/qwen2.5:0.5b 20 | sed 's/^/bifrost /'
python3 ~/ttft.py http://192.168.1.150:4000/v1/chat/completions "Bearer $KEY" qwen2.5:0.5b 20 | sed 's/^/litellm /'
The second script opens N streams at once, repeats that for a number of rounds and writes the raw timings to /tmp:
vim ~/ttft_concurrent.py
Paste the concurrent script.
#!/usr/bin/env python3
"""Concurrent streaming TTFT through an OpenAI-compatible endpoint.
usage: ttft_concurrent.py <label> <url> <auth-header> <model> <streams> [rounds]
Opens <streams> simultaneous streaming chats, <rounds> times, and reports TTFT and total time."""
import json, math, statistics, sys, threading, time, urllib.request
label, url, auth, model, streams = sys.argv[1:6]
streams = int(streams); rounds = int(sys.argv[6]) if len(sys.argv) > 6 else 2
ttfts, totals, errors = [], [], 0
lock = threading.Lock()
def one(i):
global errors
body = json.dumps({
"model": model, "stream": True, "max_tokens": 32, "temperature": 0,
"messages": [{"role": "user", "content": f"Stream {i}: name two uses of an LLM gateway."}],
}).encode()
req = urllib.request.Request(url, data=body, method="POST",
headers={"Content-Type": "application/json", "Authorization": auth})
t0 = time.perf_counter(); first = None
try:
with urllib.request.urlopen(req, timeout=300) as r:
for raw in r:
line = raw.decode().strip()
if not line.startswith("data:") or line == "data: [DONE]":
continue
delta = (json.loads(line[5:]).get("choices") or [{}])[0].get("delta", {}).get("content")
if delta and first is None:
first = time.perf_counter() - t0
except Exception:
with lock:
errors += 1
return
with lock:
ttfts.append(first * 1000); totals.append((time.perf_counter() - t0) * 1000)
for rnd in range(rounds):
threads = [threading.Thread(target=one, args=(rnd * streams + i,)) for i in range(streams)]
[t.start() for t in threads]; [t.join() for t in threads]
def p95(v):
return sorted(v)[math.ceil(len(v) * 0.95) - 1] # nearest-rank
with open(f"/tmp/ttft-{label}-{streams}.txt", "w") as f:
f.write("\n".join(f"{a:.1f} {b:.1f}" for a, b in zip(ttfts, totals)) + "\n")
print(f"{label} streams={streams} n={len(ttfts)} errors={errors} ttft_p50={statistics.median(ttfts):.0f}ms "
f"ttft_p95={p95(ttfts):.0f}ms total_p50={statistics.median(totals):.0f}ms total_p95={p95(totals):.0f}ms")
Run Ollama on the load box with eight parallel slots, pull the model, then run each concurrency level direct and through the gateway. The Bifrost run below assumes it was started with an ollama provider (an ollama_key_config URL plus allow_private_network), and the LiteLLM run with an ollama_chat/qwen2.5:0.5b model entry:
# https://github.com/ollama/ollama/releases/tag/v0.12.6
sudo docker run -d --name ollama --network host -e OLLAMA_NUM_PARALLEL=8 -v ollama:/root/.ollama ollama/ollama:0.12.6
sudo docker exec ollama ollama pull qwen2.5:0.5b
python3 ~/ttft_concurrent.py direct http://127.0.0.1:11434/v1/chat/completions 'Bearer x' qwen2.5:0.5b 8 3
python3 ~/ttft_concurrent.py bifrost http://192.168.1.150:8080/v1/chat/completions 'Bearer sk-bf-bench-0001' ollama/qwen2.5:0.5b 8 3
python3 ~/ttft_concurrent.py litellm http://192.168.1.150:4000/v1/chat/completions "Bearer $KEY" qwen2.5:0.5b 8 3
Repeat the three lines with 16 and 32 in place of 8. The LiteLLM line assumes a model_list entry named qwen2.5:0.5b with model: ollama_chat/qwen2.5:0.5b and api_base: http://192.168.1.118:11434, and $KEY is a virtual key from that stack.
For Step 3, the failover script fires 100 req/s for 30 s and, in kill mode, starts a second mock on port 8001 and kills it 10 s in:
vim ~/failover.sh
Paste the failover script.
#!/bin/bash
# Run on the load-generator box. usage: failover.sh <label> <url> <auth-header> <model> <kill|nokill>
# 100 req/s for 30 s. With "kill", the second mock backend on :8001 is killed at t=10 s.
set -u
label=$1; url=$2; auth=$3; model=$4; mode=$5
mkdir -p ~/results
if [ "$mode" = kill ]; then
pgrep -f 'mocker -host 0.0.0.0 -port 8001' >/dev/null || \
(nohup taskset -c 0,1 ~/bin/mocker -host 0.0.0.0 -port 8001 -latency 60 >/dev/null 2>&1 < /dev/null &)
sleep 2
(sleep 10; pkill -f 'mocker -host 0.0.0.0 -port 8001'; date +%s.%N > ~/results/"$label"-killed-at) &
fi
body=$(printf '{"model":"%s","messages":[{"role":"user","content":"Summarise the benefits of an LLM gateway in one line."}]}' "$model" | base64 -w0)
printf '{"method":"POST","url":"%s","body":"%s","header":{"Content-Type":["application/json"],"Authorization":["%s"]}}\n' \
"$url" "$body" "$auth" > /tmp/target-fo.json
taskset -c 2,3 ~/bin/vegeta attack -format=json -targets=/tmp/target-fo.json -rate=100/s -duration=30s -timeout=30s > ~/results/"$label".bin
~/bin/vegeta report -type=json < ~/results/"$label".bin > ~/results/"$label".json
~/bin/vegeta encode --to json < ~/results/"$label".bin > ~/results/"$label".jsonl
jq -r --arg l "$label" '"\($l) ok=\(.success*100|.*10|floor/10)% p50=\(.latencies["50th"]/1e6|.*10|floor/10)ms p99=\(.latencies["99th"]/1e6|.*10|floor/10)ms max=\(.latencies.max/1e6|floor)ms codes=\(.status_codes|tostring)"' ~/results/"$label".json
The 429 test needs the mock restarted with one key that always gets rate limited. Stop the running mock first, or the new one cannot bind port 8000 and the test quietly runs against a mock that never returns 429. On the load box:
pkill -x mocker; sleep 1
nohup taskset -c 0,1 ~/bin/mocker -host 0.0.0.0 -port 8000 -latency 60 -rate-limited-keys "Bearer mock-key-429" > ~/mocker.log 2>&1 < /dev/null &
On the gateway box, LiteLLM lists two deployments under each test model and adds the router retry settings:
vim ~/harness/litellm/config-failover.yaml
Paste the LiteLLM failover config.
model_list:
# one key always gets 429 from the mock, the other succeeds
- model_name: rl/gpt-4o-mini
litellm_params:
model: openai/gpt-4o-mini
api_base: http://192.168.1.118:8000/v1
api_key: mock-key-429
- model_name: rl/gpt-4o-mini
litellm_params:
model: openai/gpt-4o-mini
api_base: http://192.168.1.118:8000/v1
api_key: mock-key-a
# two backends; the one on :8001 is killed mid-run
- model_name: ha/gpt-4o-mini
litellm_params:
model: openai/gpt-4o-mini
api_base: http://192.168.1.118:8000/v1
api_key: mock-key-a
- model_name: ha/gpt-4o-mini
litellm_params:
model: openai/gpt-4o-mini
api_base: http://192.168.1.118:8001/v1
api_key: mock-key-a
litellm_settings:
set_verbose: false
json_logs: true
request_timeout: 600
router_settings:
routing_strategy: simple-shuffle
redis_host: 127.0.0.1
redis_port: 6379
num_retries: 2
allowed_fails: 1
cooldown_time: 60
general_settings:
master_key: os.environ/LITELLM_MASTER_KEY
database_url: os.environ/DATABASE_URL
proxy_batch_write_at: 60
database_connection_pool_limit: 10
disable_error_logs: true
use_redis_transaction_buffer: true
Bifrost gets three providers (the two-key rl provider and the mock/mock2 pair) and two virtual keys, one per test:
vim ~/harness/bifrost/config-failover.json
Paste the Bifrost failover config.
{
"$schema": "https://www.getbifrost.ai/schema",
"client": {
"initial_pool_size": 5000,
"drop_excess_requests": false,
"enable_logging": true
},
"providers": {
"rl": {
"keys": [
{ "name": "key-429", "value": "mock-key-429", "models": ["gpt-4o-mini"], "weight": 1.0 },
{ "name": "key-ok", "value": "mock-key-a", "models": ["gpt-4o-mini"], "weight": 1.0 }
],
"network_config": { "base_url": "http://192.168.1.118:8000", "allow_private_network": true, "max_retries": 2 },
"custom_provider_config": { "base_provider_type": "openai", "allowed_requests": { "chat_completion": true, "chat_completion_stream": true } }
},
"mock": {
"keys": [ { "name": "a", "value": "mock-key-a", "models": ["gpt-4o-mini"], "weight": 1.0 } ],
"network_config": { "base_url": "http://192.168.1.118:8000", "allow_private_network": true },
"custom_provider_config": { "base_provider_type": "openai", "allowed_requests": { "chat_completion": true, "chat_completion_stream": true } }
},
"mock2": {
"keys": [ { "name": "b", "value": "mock-key-a", "models": ["gpt-4o-mini"], "weight": 1.0 } ],
"network_config": { "base_url": "http://192.168.1.118:8001", "allow_private_network": true },
"custom_provider_config": { "base_provider_type": "openai", "allowed_requests": { "chat_completion": true, "chat_completion_stream": true } }
}
},
"governance": {
"virtual_keys": [
{
"id": "vk-rl", "name": "rl", "value": "sk-bf-rl-0001", "is_active": true,
"provider_configs": [ { "provider": "rl", "weight": 1.0, "allowed_models": ["*"], "key_ids": ["*"] } ]
},
{
"id": "vk-ha", "name": "ha", "value": "sk-bf-ha-0001", "is_active": true,
"provider_configs": [
{ "provider": "mock", "weight": 0.5, "allowed_models": ["*"], "key_ids": ["*"] },
{ "provider": "mock2", "weight": 0.5, "allowed_models": ["*"], "key_ids": ["*"] }
]
}
]
},
"config_store": { "enabled": true, "type": "sqlite", "config": { "path": "/app/data/config.db" } },
"logs_store": { "enabled": true, "type": "sqlite", "config": { "path": "/app/data/logs.db" } }
}
Start one gateway at a time with its failover config. The Compose file reads LITELLM_CONFIG, which has to sit after sudo or sudo drops it:
cd ~/harness/litellm && sudo LITELLM_CONFIG=config-failover.yaml docker compose up -d
Generate a key as before and run the two LiteLLM lines below from the load box. Then stop the LiteLLM stack and start Bifrost with its failover config:
cd ~/harness/litellm && sudo docker compose down
sudo docker rm -f bifrost
sudo rm -rf ~/bf-data; mkdir -p ~/bf-data && cp ~/harness/bifrost/config-failover.json ~/bf-data/config.json
sudo docker run -d --name bifrost --cpuset-cpus 0-2 --network host -e BIFROST_SETUP_TOKEN=lab-setup-token -v ~/bf-data:/app/data maximhq/bifrost:v2.4.0
until curl -sf localhost:8080/health >/dev/null; do sleep 0.2; done
These are the four failover runs, LiteLLM first, then Bifrost:
~/failover.sh litellm-429 http://192.168.1.150:4000/v1/chat/completions "Bearer $KEY" rl/gpt-4o-mini nokill
~/failover.sh litellm-kill http://192.168.1.150:4000/v1/chat/completions "Bearer $KEY" ha/gpt-4o-mini kill
~/failover.sh bifrost-429 http://192.168.1.150:8080/v1/chat/completions 'Bearer sk-bf-rl-0001' rl/gpt-4o-mini nokill
~/failover.sh bifrost-kill http://192.168.1.150:8080/v1/chat/completions 'Bearer sk-bf-ha-0001' gpt-4o-mini kill
For Step 4, the MCP server is a FastMCP script with two tools, run from a virtual environment on the load box:
python3 -m venv ~/mcpvenv && ~/mcpvenv/bin/pip install -q "mcp==1.30.0"
vim ~/lab_tools.py
Paste the server.
from mcp.server.fastmcp import FastMCP
mcp = FastMCP("lab-tools", host="0.0.0.0", port=9000)
@mcp.tool()
def disk_free(path: str = "/") -> str:
"""Return free space on a path in GiB."""
import shutil
total, used, free = shutil.disk_usage(path)
return f"{path}: {free / 2**30:.1f} GiB free of {total / 2**30:.1f} GiB"
@mcp.tool()
def add(a: int, b: int) -> int:
"""Add two integers."""
return a + b
if __name__ == "__main__":
mcp.run(transport="streamable-http")
Each gateway then points at it. LiteLLM takes an mcp_servers block in its config.yaml:
mcp_servers:
lab_tools:
url: http://192.168.1.118:9000/mcp
transport: http
Bifrost takes an mcp.client_configs entry in config.json, and the virtual key needs "mcp_configs": [ { "mcp_client_name": "lab_tools", "tools_to_execute": ["*"] } ] or it lists no tools:
"mcp": {
"client_configs": [
{
"name": "lab_tools",
"connection_type": "http",
"connection_string": "http://192.168.1.118:9000/mcp",
"tools_to_execute": ["*"]
}
]
},
Last comes the probe, a real MCP client that initialises a session, lists the tools and calls add:
vim ~/mcp_probe.py
Paste the probe.
"""Connect to a gateway's MCP endpoint as a real MCP client, list tools, call one.
usage: mcp_probe.py <url> <header-name> <header-value>"""
import asyncio, sys
from mcp import ClientSession
from mcp.client.streamable_http import streamablehttp_client
async def main(url, hname, hval):
async with streamablehttp_client(url, headers={hname: hval}) as (r, w, _):
async with ClientSession(r, w) as s:
init = await s.initialize()
print("server:", init.serverInfo.name, init.serverInfo.version)
tools = await s.list_tools()
print("tools:", ", ".join(t.name for t in tools.tools))
add = next(t.name for t in tools.tools if t.name.endswith("add"))
res = await s.call_tool(add, {"a": 2, "b": 3})
print(f"call {add}(2, 3) ->", res.content[0].text)
asyncio.run(main(*sys.argv[1:4]))
In the lab these blocks lived in their own files, config-mcp.yaml and config-mcp.json (the Ollama configs plus the MCP blocks), and each gateway was started fresh with them: sudo LITELLM_CONFIG=config-mcp.yaml docker compose up -d for LiteLLM, and the remove, copy and run block above (with config-mcp.json) for Bifrost. Editing a mounted config is not enough on its own, because neither gateway reloads it without a restart. Start the MCP server on the load box, then probe each gateway with the header it expects. With mcp 1.30.0 the probe may print a DeprecationWarning for streamablehttp_client; the call still completes:
nohup ~/mcpvenv/bin/python ~/lab_tools.py > ~/mcp.log 2>&1 < /dev/null &
~/mcpvenv/bin/python ~/mcp_probe.py http://192.168.1.150:4000/mcp x-litellm-api-key "Bearer $SCOPED_KEY"
~/mcpvenv/bin/python ~/mcp_probe.py http://192.168.1.150:8080/mcp Authorization 'Bearer sk-bf-bench-0001'
$SCOPED_KEY is a LiteLLM key created with "object_permission": {"mcp_servers": ["lab_tools"]} in the /key/generate body; a key without that grant gets HTTP 403.
Running the LiteLLM side with more workers is the obvious next experiment. On a box with 16 cores and 16 workers, the per-core rate measured here predicts roughly 560 to 880 req/s, still below what Bifrost served on three. On Proxmox, set the VM CPU type to host before you start, or the LiteLLM image will not run at all.
Where the vendor claims held up and where they did not
Every LiteLLM vs Bifrost comparison we found in November 2026 was published by Maxim, the company behind Bifrost, so each of its claims was checked against what this rig measured:
- “LiteLLM fails at 500 RPS” (held up): the Maxim post reports 88.78% success at 500 req/s on a 2 vCPU t3.medium, against a real OpenAI key rather than a mock. Tuned LiteLLM on three cores here managed 45% at 500 req/s, and the single-worker default only 77% at 100 req/s.
- “11 µs overhead at 5,000 RPS” (not reproduced): that figure is Bifrost’s internal processing time, which its docs say excludes JSON handling and the HTTP call. End to end through a second VM, Bifrost added 2.2 to 3.9 ms and topped out at 1,995 req/s on three cores of an older CPU.
- “LiteLLM has no MCP gateway” (wrong): the open source LiteLLM image served
/mcpto a real MCP client, listed tools and executed one, with per-key scoping. - “Bifrost uses far less memory” (held up at idle, not under overload): 128 to 319 MiB against 2.1 GiB at idle, but 6.0 GiB against 2.7 GiB while queuing past its ceiling.
- “Docker size 80 MB against over 700 MB” (direction held, numbers did not): the Bifrost image is 104 MB compressed and 412 MB on disk, and the LiteLLM database image 403 MB compressed and 1.72 GB on disk.
LiteLLM’s own benchmark page deserves the same scrutiny. It reports 1,170 req/s across four instances (4 vCPU, 8 GB and 4 workers each) with 2 ms of median overhead, about 290 req/s per instance. That overhead is read from the x-litellm-overhead-duration-ms response header, so it is measured inside the proxy, the way the Bifrost 11 µs figure is. End to end on this CPU we measured roughly 35 to 55 req/s per core and 30 to 55 ms of median overhead. The honest summary is that Bifrost is an order of magnitude cheaper to run per request, and LiteLLM gives more of its governance features away. Pick on the constraint you actually have.