AI

LiteLLM vs Bifrost: Independent LLM Gateway Benchmark

On the same three CPU cores, Bifrost served up to 1,540 requests per second with logging on, and a tuned LiteLLM proxy topped out near 110. That gap is real, but it is not the whole LiteLLM vs Bifrost story. LiteLLM ships the MCP gateway that the vendor comparison pages say it lacks, and both gateways survived a killed backend without a single failed request.

Original content from computingforgeeks.com - post 183812

The throughput numbers come from a re-run of the mock-server benchmark that Bifrost publishes, on two Proxmox VMs. Each gateway was pinned to cores 0-2, and LiteLLM got workers, Postgres and Redis instead of the single-worker default. At 100 requests per second Bifrost added 2.4 ms at the median and LiteLLM 30 to 55 ms (LiteLLM v1.106.6 and Bifrost v2.4.0, benchmarked November 2026 on Ubuntu 24.04). The streaming test ran through Ollama on CPU because no GPU was available for vLLM. The enterprise rows of the matrix come from vendor docs and dashboards, not from a licensed install.

LiteLLM vs Bifrost results at a glance

Bifrost wins every throughput and footprint row by a wide margin, and the two are level on failover, MCP and streaming within the noise of a CPU model. The table holds the measured values. Each row links back to the section that explains how it was taken.

Measured on the test boxBifrost (logging on)LiteLLM (3 workers, Postgres, Redis)
Added p50 latency at 100 req/s, vs the mock alone+2.4 ms+30.5 to +54.9 ms (two passes)
Highest offered rate with 100% success before queueing1,000 req/s100 req/s
Peak served throughput, logging on / off1,540 / 1,995 req/s110 / 161 req/s
Streaming TTFT p50 through Ollama (direct: 254 ms)274 ms290 ms
Key that always gets HTTP 429, 3,000 requests100% success, p99 661 ms100% success, p99 1,283 ms
Backend killed 10 s into a 30 s run100% success, p99 66 ms100% success, p99 179 ms
MCP gateway in the open source imageYes, testedYes, tested
Fresh start to healthy12.7 to 20.9 s44.7 to 112.8 s
Idle memory319 MB (128 MB without log store)2.1 GB plus 54 MB Postgres
Image size on disk412 MB1.72 GB
Licence of the gateway codeApache-2.0MIT, except the proprietary enterprise/ directory

The throughput rows decide the choice only when the gateway itself is the bottleneck, which is rarer than the vendor charts suggest. A real model at 100 req/s needs a GPU cluster. On our CPU model the gap shrank to 16 ms of TTFT, roughly 6% of the first-token time. The picks below follow from that:

  • A small team on one box: Bifrost. One static binary in a 412 MB image, 128 to 319 MB at idle and no database to run, so it fits beside the model server.
  • A platform team that needs per-team budgets and SSO without a contract: LiteLLM. Teams, budgets, guardrails and the admin UI all load in the open source image, and SSO is free for up to 5 users.
  • Thousands of requests per second through one gateway: Bifrost, or LiteLLM with far more cores. On this CPU one LiteLLM worker handled 35 to 55 req/s, so 1,000 req/s means roughly 20 to 30 workers.
  • A Python shop that extends the gateway: LiteLLM. Its callbacks and guardrail hooks are Python, and tools such as Aider already use the same library underneath.

Bifrost gates clustering, guardrails, RBAC, audit logs and user management behind its enterprise licence, and the dashboard says so on each page. LiteLLM gates organizations, audit logs and SSO past 5 users. Neither split is hidden, but they fall in different places, so check the matrix further down against the features you need before you trust the throughput row.

How the test rig was built

The rig follows the method in maximhq/bifrost-benchmarking: a Go mock provider with a fixed 60 ms latency stands in for OpenAI, and Vegeta fires a constant request rate at the gateway. Using the vendor tool removes the argument that the harness favours one side. The difference is in the gateway setup, which the vendor repo does not publish for LiteLLM.

Benchmark rig diagram: Vegeta on bench02 drives Bifrost or LiteLLM on bench01, which forward to a 60 ms mock on bench02

Both VMs ran on one Proxmox node with an Intel Core i7-6700 (4 cores, 8 threads) that was otherwise below 5% CPU. Only one gateway ran at a time. How each number was taken:

  • Pinning: the gateway container got --cpuset-cpus 0-2 on bench01, and the LiteLLM Postgres and Redis containers got core 3. On bench02 the mock ran under taskset -c 0,1 and Vegeta under taskset -c 2,3.
  • Latency and throughput: the Vegeta JSON report for a 30 s constant-rate run with a 30 s timeout, 10 s apart. Overhead is the gateway p50 minus the p50 of the same ladder fired straight at the mock.
  • Auth on both: every request carried a virtual key, because Bifrost enforces one by default and LiteLLM refuses to start without a master key.
  • Memory and CPU: docker stats sampled every 3 to 5 s during each ladder. Idle memory is the reading 20 s after the health check passed.
  • Start time: wall clock from docker run or docker compose up until /health or /health/liveliness answered.

LiteLLM got what its production best practices page asks for, on the litellm-database image: one worker per vCPU (3), Postgres for keys and spend, Redis for the transaction buffer, proxy_batch_write_at: 60, a pool limit of 10 and error logs off. Two caveats on that tuning. The page sets workers equal to vCPUs only for a single VM (on Kubernetes it runs one worker per pod and scales pods), and it documents the Redis transaction buffer for roughly 1,000 req/s or ten-plus instances, far above this rig, but we enabled it anyway. The multi-instance setup it describes was not tested, because the lab allowed three cores per gateway, so any figure above one instance in this article is a projection. A default single-worker run is in the results too, because that is the setup the vendor comparisons appear to use. The images that ran, with their digests:

maximhq/bifrost:v2.4.0 sha256:a1c90607dcfd13c1fb2f9476ceb3bb24407a50ade7fdf3e61a59fb42b09e5c57 412MB
ghcr.io/berriai/litellm:v1.106.6 sha256:37e2dafb088a7723beb998590682957386dfc83ffea1848486f8e0e5cfac0b06 1.81GB
ghcr.io/berriai/litellm-database:v1.106.6 sha256:7ef0e78fe00ad5ea7ea45666cc1d1f02cb7076aee69091543bf397de52db33ce 1.72GB
redis:7-alpine sha256:8da19c7399dca900c6822a1cf30f9d2285585e2c4b457ff4095b0f4500054d3f 57.8MB
postgres:16-alpine sha256:0cefaabd882e42bcb2a2f3dc56d3ce4770560ba62fd7f197f1a2e008fd5731ab 420MB

Four things broke while the rig was being built, and each one cost time that a reader setting up either gateway will also spend. They are listed here with the exact text, because the messages are what people search for.

Bifrost: “connection to private IP 192.168.1.118 is not allowed”

Bifrost blocks provider URLs on private networks by default, so a mock, an Ollama box or a vLLM server on the LAN returns HTTP 502 with provider_connection_failed. The model list fails the same way at start, which surfaces first as the less helpful no keys found for provider. The fix is "allow_private_network": true in the provider network_config. A second trap sits next to it: the built-in openai provider ignores base_url, so a self-hosted OpenAI-compatible backend needs a custom provider with base_provider_type: openai.

LiteLLM: “Fatal glibc error: CPU does not support x86-64-v2”

The LiteLLM image exited at once on a Proxmox VM using the default kvm64 CPU type, while the Bifrost container on the same VM ran fine. Setting the VM CPU type to host and restarting it fixed it. Any hypervisor that exposes a baseline x86-64 CPU model will hit this.

LiteLLM: “use_redis_transaction_buffer is enabled in general_settings but no Redis is configured”

The production page recommends use_redis_transaction_buffer, and router_settings.redis_host does not satisfy it. The proxy refused to start until REDIS_HOST and REDIS_PORT were set as environment variables. The error offers two fixes, a Redis response cache or the REDIS_* variables. We took the variables, because a cache would answer identical benchmark requests without calling the mock.

LiteLLM: “proxy refused to start: no master key is set”

This LiteLLM release will not boot without a master key unless you set LITELLM_DANGEROUSLY_PERMIT_WEAK_OR_UNSET_MASTER_KEY=true. So the open, keyless proxy that older comparisons benchmarked is no longer the default, and every LiteLLM run here used the master key or a generated virtual key.

Step 1: Measure gateway overhead against a 60 ms mock

Straight at the mock, Vegeta held a p50 between 60.6 and 60.9 ms at every rate from 100 to 5,000 req/s. The one blemish was 127 failed requests at 3,000 req/s, because localhost resolved to ::1 and the mocker listens on IPv4 only. Bifrost added 2.2 to 3.9 ms at the median up to 1,000 req/s, with all 30,000 requests at 1,000 req/s returning 200:

Vegeta load ladder against Bifrost v2.4.0 showing p50 latency of 63 to 65 ms up to 1000 requests per second against a 60 ms mock

Rows above 1,000 req/s in that capture say more about the load box than about Bifrost. Status 0 is the Vegeta code for any transport error, and here the errors were bind: address already in use on bench02: the client ran out of ephemeral ports. It sent only about 1,050 to 1,130 req/s of the 3,000 and 5,000 asked for. After widening the port range on the load box (Step 6), the same config held 1,500 req/s with every request successful and a p50 of 152 ms. At 2,000 and 3,000 offered it served 1,499 to 1,540 req/s while requests queued in its buffer. Bifrost did not fail requests past its ceiling, it queued them. The tuned LiteLLM stack hit the same wall at about one fourteenth of that rate:

Vegeta load ladder against tuned LiteLLM v1.106.6 with three workers, Postgres and Redis, saturating near 110 requests per second

At 100 req/s every LiteLLM request succeeded with a p50 of 115.8 ms, which is 55 ms of gateway time on top of the 60 ms mock. At 250 req/s success fell to 80%, with 144 HTTP 503 responses and 1,303 status 0 errors. The more useful finding is what happens past saturation: served throughput dropped from 110 req/s at 500 offered to 30 to 48 req/s at 1,000 and above, with connection reset by peer and client timeouts as the errors. Bifrost kept serving its ceiling under overload, while LiteLLM collapsed, so a LiteLLM deployment needs its rate limit set below its measured ceiling rather than at it.

Separating out logging changes both numbers. With disable_spend_logs: true, the LiteLLM ceiling rose from about 110 to 161 req/s. Its p50 at 100 req/s also fell from 115.8 to 91.4 ms in that pass, but a later run with spend logs on measured the same 91.4 ms (its p99 was 832 ms, against 318 ms for the first spend-logs-on run). The median gap is run-to-run drift. The ceiling is the real cost of spend logging. With the log store and request logging off, the Bifrost p50 at 1,000 req/s barely moved (64.6 to 63.1 ms), but its ceiling rose from about 1,540 to 1,995 req/s. Logging is not free in either product. In Bifrost the default SQLite log store is what caps throughput first:

ConfigurationOffered req/sSuccessServed req/sp50p99
Mock only (baseline)1,000100%99760.6 ms62.2 ms
Bifrost, logging on1,000100%99764.6 ms175.6 ms
Bifrost, logging on, tuned client1,500100%1,490152.4 ms336.4 ms
Bifrost, logging off, tuned client2,000100%1,99573.7 ms518.2 ms
LiteLLM tuned, spend logs on (first pass)100100%99115.8 ms318.0 ms
LiteLLM tuned, spend logs off100100%9991.4 ms144.5 ms
LiteLLM tuned, spend logs off150100%146880.4 ms2,279 ms
LiteLLM default, 1 worker, no DB50100%492,381 ms6,850 ms

The last row is the setup most published comparisons describe. One LiteLLM worker already queued requests at 25 req/s (p95 7.6 s), which is why the vendor charts show multi-second LiteLLM latencies. Three workers on three cores lifted the clean ceiling to about 100 req/s and the peak from 49 to 110 req/s, so the tuning roughly doubled throughput and the per-core rate stayed near 35 to 55 req/s. CPU, not configuration, is what LiteLLM runs out of: docker stats showed its three pinned cores saturated in every overloaded step, and the Bifrost cores were saturated the same way above 1,500 req/s.

Step 2: Measure streaming TTFT through a real model

A mock proves overhead, but users feel time to first token. The streaming test sent 20 sequential requests (after one warm-up) for qwen2.5:0.5b on Ollama, running on the four CPU cores of bench02, and timed the first SSE chunk that carried content. We would have run this on vLLM, as in the vLLM production install, but the GPU budget for this run was zero. Ollama on CPU stood in as the real backend. The setup matches the Ollama install guide, in a container.

direct runs=20 ttft_p50=253.9ms ttft_min=234.0ms ttft_max=314.6ms total_p50=2159.3ms
bifrost runs=20 ttft_p50=273.7ms ttft_min=240.2ms ttft_max=353.3ms total_p50=2237.0ms
litellm runs=20 ttft_p50=289.8ms ttft_min=245.9ms ttft_max=781.5ms total_p50=2156.1ms
direct-rerun runs=20 ttft_p50=246.1ms ttft_min=219.5ms ttft_max=294.5ms total_p50=2184.5ms

Bifrost added about 20 to 28 ms to TTFT and LiteLLM about 36 to 44 ms, depending on which direct run you take as the baseline. That is more than the Bifrost overhead on the mock, because both gateways went through their own Ollama provider (ollama_chat/ in LiteLLM, ollama in Bifrost) while the direct run used the Ollama OpenAI endpoint. The format translation is part of the number. The total time to finish 64 tokens stayed within 81 ms across all four runs, inside the run-to-run noise of a CPU model. On a GPU serving tens of tokens per second the gap would matter more for TTFT and less for total time. The Ollama vs vLLM vs llama.cpp results show how much the backend itself moves those numbers.

Sequential requests flatter every gateway, so a second pass ran 8, 16 and 32 simultaneous streams (3 rounds each, 32 tokens per reply) with Ollama set to OLLAMA_NUM_PARALLEL=8. This pass used fresh clones of the same template with the same image digests, and each concurrency level ran direct, through Bifrost and through LiteLLM back to back:

direct streams=8 n=24 errors=0 ttft_p50=1419ms ttft_p95=1780ms total_p50=5359ms total_p95=5387ms
bifrost streams=8 n=24 errors=0 ttft_p50=1372ms ttft_p95=1534ms total_p50=5344ms total_p95=5350ms
litellm streams=8 n=24 errors=0 ttft_p50=1546ms ttft_p95=1764ms total_p50=5072ms total_p95=5369ms
direct streams=16 n=48 errors=0 ttft_p50=3380ms ttft_p95=6695ms total_p50=7172ms total_p95=10046ms
bifrost streams=16 n=48 errors=0 ttft_p50=3940ms ttft_p95=6861ms total_p50=7817ms total_p95=10932ms
litellm streams=16 n=48 errors=0 ttft_p50=3742ms ttft_p95=7312ms total_p50=8218ms total_p95=11140ms
direct streams=32 n=96 errors=0 ttft_p50=8914ms ttft_p95=16092ms total_p50=12995ms total_p95=20362ms
bifrost streams=32 n=96 errors=0 ttft_p50=8830ms ttft_p95=17321ms total_p50=13277ms total_p95=21996ms

The LiteLLM row at 32 streams is missing because the Proxmox node rebooted in the middle of it, with all eight host threads busy. An earlier pass of the same test, 2 rounds per level, had completed it: LiteLLM 11.6 s TTFT p50 at 32 streams, against 9.8 s direct and 10.4 s through Bifrost in that pass. Read together, the two passes say the backend queue dominates once streams outnumber the model’s parallel slots.

At 8 and 16 streams the gateways landed within 560 ms of direct in either direction, which is inside the run-to-run spread (direct alone moved from 3.9 to 3.4 s at 16 streams between passes). Only at 32 streams did LiteLLM fall behind, by about 1.8 s at the median in the one pass that finished it, which fits a Python gateway with three workers juggling 32 open SSE streams. Treat that single point as a lead to re-test on a GPU backend, not as a result.

Step 3: Break a key and kill a backend

Both gateways passed both failure tests with zero failed requests at 100 req/s, the rate LiteLLM could sustain. The mock was restarted with -rate-limited-keys "Bearer mock-key-429", so one of two upstream keys answered every call with HTTP 429. For the second test, a second mock on port 8001 sat behind the same model name and was killed 10 s into a 30 s run:

Failover test results for LiteLLM and Bifrost with a key that always gets 429 and a backend killed mid-run, all 3000 requests returning 200

LiteLLM handled both through its router: two deployments under one model_name, num_retries: 2, allowed_fails: 1 and a 60 s cooldown. Its p99 on the 429 test was 1,283 ms because a request that drew the bad key paid for a retry. Bifrost needed no retry settings beyond max_retries on the 429 provider. On the kill test its virtual key split traffic 50/50 across two providers, and after the kill every request was ultimately served by the survivor, with a p99 of only 65.5 ms.

Bifrost log counters explain that low p99, and they also show a weakness. After both runs Bifrost reported 6,978 upstream attempts for 6,000 client requests, an attempt success rate of 85.98% against a user-facing success rate of 100%. The provider behind the killed mock logged 1,467 attempts, close to half of the 3,000 requests, so Bifrost never stopped choosing it. For the last 20 s about every second request tried the dead backend first and was retried on the survivor, which logged 2,511.

That cost almost nothing here because a closed port refuses at once. We did not test a backend that hangs instead, but on this evidence it would add the provider timeout to about every second request, and the Bifrost circuit breaker is an enterprise feature that its docs describe as triggered by response header signals. In LiteLLM, allowed_fails and cooldown_time put a failing deployment on cooldown in the open source build, although a hung call still waits for request_timeout (600 s in this config) before it counts as a failure, so set that timeout low on a router you rely on for failover.

Step 4: Test the MCP gateway claim

The Maxim feature-by-feature comparison says “LiteLLM does not provide a native MCP gateway”. That is wrong for the open source LiteLLM image we ran. A small FastMCP server with two tools was added to both gateway configs. A real MCP client (the Python mcp SDK over streamable HTTP) connected to the /mcp endpoint of each gateway, listed the tools and called one:

MCP client listing and calling tools through the LiteLLM and Bifrost MCP gateway endpoints

Both gateways prefix tool names with the server name (lab_tools-add), and both deny tools to keys by default, which is the right default. A fresh LiteLLM virtual key got HTTP 403 with “The key has no MCP servers granted” until it was created with object_permission.mcp_servers. The Bifrost virtual key listed zero tools until its mcp_configs named the client. Per-key tool scoping therefore works in both open source images. Virtual MCPs in Bifrost, which bundle tools from several servers into one endpoint, are an enterprise feature according to its docs, and we did not test them.

Step 5: Compare cold start, image size and memory

Bifrost reached a healthy /health in a median of 15.6 s across ten fresh starts (12.7 to 20.9 s), and its own log put bootstrap at 12.6 s. That is slow for a Go binary, and the start log is mostly config-store and log-store migrations. LiteLLM took 112.8 s on its first start against an empty Postgres and 44.7 to 89.0 s on later fresh stacks. It needed 35.8 s to restart the proxy alone against an existing database. Without a database, the single-worker image took 70.7 s, so most of the LiteLLM start time is not the database.

FootprintBifrostLiteLLM
Image on disk (compressed)412 MB (104 MB)1.72 GB (403 MB), database image
Idle memory128 MB without log store, 319 MB with it541 MB with 1 worker, 2.1 GB with 3 workers
Peak memory during the ladder4.0 GiB without log store, 6.0 GiB with it2.7 GiB proxy, 302 MiB Postgres, 6 MiB Redis
Average CPU at 100 req/s27% of one core202% (2.0 cores), plus 4% Postgres and 3% Redis
Average CPU at 1,000 req/s213% (2.1 cores)Not reachable (ceiling about 110 req/s)
Extra services requiredNone (SQLite inside the volume)Postgres for keys and spend, Redis for multi-worker

The CPU rows average the docker stats samples (one every 3 to 5 s) taken while a dedicated 30 s run was under load, in the second lab pass with the spend-logs-on LiteLLM config. That LiteLLM run measured a 91.4 ms p50 at 100 req/s, 24 ms better than the first pass, which is the size of the run-to-run drift on this stack. The CPU rows explain the throughput gap better than any latency figure: Bifrost at 1,000 req/s used about the same CPU as LiteLLM at 100 req/s, so per request LiteLLM spends 7.5 to 9.5 times the compute. The peak memory row is the one that would worry us in production. Bifrost grew to 6.0 GiB while it queued requests above its ceiling, with up to about 20,000 client connections open, a 10,000-deep provider buffer and drop_excess_requests: false in the config. On a host with less RAM than that, overload would end in the OOM killer rather than in slow responses. We did not re-run with dropping enabled, but we would set drop_excess_requests: true and size buffer_size to the RAM available. LiteLLM stayed within 0.7 GB of its idle memory under overload because it rejected and timed out requests instead.

Open source vs enterprise feature matrix

The matrix marks where each claim comes from, because only part of it could be tested without a licence. “Lab” means it ran in this benchmark. “UI” means the page opened in the open source dashboard, or showed a licence notice. “Docs” means it is taken from the vendor enterprise or feature pages and was not run:

FeatureBifrost open sourceLiteLLM open source
Virtual keysYes (lab)Yes (lab)
TeamsPage opens (UI)Yes (UI)
Budgets and rate limitsYes (docs)Yes (UI, docs)
Fallbacks and load balancingYes (lab)Yes (lab)
Exact response cachingYes (UI)Yes, Redis (UI)
Semantic cachingYes (docs)Yes, redis-semantic, qdrant-semantic or valkey-semantic (docs)
GuardrailsEnterprise (UI)Presidio and custom guardrails open, several integrations enterprise (docs)
MCP gateway with per-key toolsYes (lab); Virtual MCPs enterprise (docs)Yes (lab)
SSOEnterprise, OIDC and SCIM (UI, docs)Free up to 5 users, licence beyond (docs)
Prometheus and OpenTelemetryYes (UI)Yes (docs)
Request logs and usage UIYes (lab)Yes (lab)
RBAC, users, audit logsEnterprise (UI)Audit logs and RBAC enterprise (docs)
Organizations, business unitsEnterprise (UI)Enterprise (UI)
Clustering and adaptive routingEnterprise (UI); multinode via shared config (docs)Multiple instances on shared Postgres and Redis (docs)

The docs-sourced rows rest on these vendor pages, read in November 2026:

The licences follow the same split. The Bifrost repository is Apache-2.0 and its enterprise build ships as a separate private image. The LiteLLM root LICENSE is MIT except for the enterprise/ directory, which needs a subscription for production use, and that is why GitHub reports the licence as NOASSERTION. The practical difference is the governance layer: LiteLLM gives away more of it in the open image, while Bifrost gives away more raw performance.

The two dashboards side by side

Both gateways ship a web UI in the same container, and both logged the six Ollama requests sent to them after the benchmark. The Bifrost LLM Logs page puts request count, success rate, average latency and token totals above the log table:

Bifrost LLM Logs page showing six chat requests routed to Ollama with 100 percent success

Bifrost v2.4.0 also locks its management API until you either create an admin account or send the X-Bifrost-Setup-Token header, and the dashboard keeps a warning in the sidebar until you do. The LiteLLM request log shows duration and a TTFT column per call, which is useful for spotting a slow backend without leaving the UI:

LiteLLM Admin UI request logs with duration and TTFT columns for calls routed to Ollama

In the red banner, LiteLLM warns that anyone holding the master key can log in as admin. It asks you to create a real admin account and then set disable_env_credential_login: true, and it is worth doing before the UI is reachable from anywhere. The MCP server defined in config.yaml showed up as healthy on the MCP Servers page:

LiteLLM MCP Servers page listing the lab_tools server loaded from config.yaml as healthy

Pages like these are where the enterprise boundaries in the matrix come from. The open source Bifrost dashboard lists Cluster, Guardrails, Adaptive Routing and the user management pages in the sidebar, then shows a licence notice when you open them:

Bifrost open source dashboard showing the cluster page locked behind the enterprise licence

Of the twelve LiteLLM pages we opened, only Organizations carried a licence notice. Logs, MCP servers, guardrails, policies, teams, budgets, usage, caching, access groups, internal users and Lens all loaded without a key:

LiteLLM Organizations page stating it is an enterprise feature that needs a licence key

A page that loads is not proof that every option on it works without a licence, which is why the matrix marks those rows as UI rather than lab.

Step 6: Reproduce the benchmark

You need two Linux VMs with 4 vCPUs each, Docker and a CPU type that exposes x86-64-v2. Replace 192.168.1.150 (the gateway box) and 192.168.1.118 (the load box) with your own addresses in the files below. Every file and script below is the one the numbers came from, printed in full because the harness is not in a public repository yet. Install the tools on both VMs:

sudo apt-get update -qq
sudo apt-get install -y -qq docker.io docker-compose-v2 jq sysstat bc git python3-venv
sudo usermod -aG docker $USER
sudo systemctl disable --now unattended-upgrades apt-daily.timer apt-daily-upgrade.timer

Disabling the upgrade timers matters more than it looks. On our first pass, apt upgraded packages on both boxes in the middle of a ladder and restarted sshd, and that run had to be thrown away. On the load box, also widen the ephemeral port range, because Vegeta opens a new connection for every request it cannot reuse and ran out of ports above about 1,000 req/s. The setting does not survive a reboot, and even with it the client still hit a few port errors at 5,000 req/s offered:

sudo sysctl -w net.ipv4.ip_local_port_range="1024 65535" net.ipv4.tcp_tw_reuse=1

On the load box, build the mocker, the Bifrost benchmark tool and the Vegeta CLI from source inside a Go container, so nothing but Docker has to be installed on the host:

git clone -q https://github.com/maximhq/bifrost-benchmarking.git ~/bifrost-benchmarking && cd ~/bifrost-benchmarking
# https://github.com/tsenart/vegeta/releases/tag/v12.12.0
sudo docker run --rm -v $PWD:/src -v $HOME/bin:/out -w /src -e CGO_ENABLED=0 -e GOFLAGS=-buildvcs=false golang:1.24 sh -c "cd mocker && go build -o /out/mocker . && cd .. && go build -o /out/benchmark benchmark.go && GOBIN=/out go install github.com/tsenart/vegeta/[email protected]"

Start the mock on cores 0 and 1 with a fixed 60 ms latency. The -host 0.0.0.0 flag matters, because the mocker binds to localhost by default and the gateway box would get connection refused:

nohup taskset -c 0,1 ~/bin/mocker -host 0.0.0.0 -port 8000 -latency 60 > ~/mocker.log 2>&1 < /dev/null &

On the gateway box, create the Bifrost config. The custom mock provider, allow_private_network and the virtual key declared in the file are the three parts that make it work against a LAN backend:

mkdir -p ~/harness/bifrost ~/harness/litellm
vim ~/harness/bifrost/config.json

Paste the configuration:

{
  "$schema": "https://www.getbifrost.ai/schema",
  "client": {
    "initial_pool_size": 5000,
    "drop_excess_requests": false,
    "enable_logging": true
  },
  "providers": {
    "mock": {
      "keys": [
        { "name": "mock-a", "value": "mock-key-a", "models": ["gpt-4o-mini"], "weight": 1.0 }
      ],
      "network_config": {
        "base_url": "http://192.168.1.118:8000",
        "default_request_timeout_in_seconds": 60,
        "allow_private_network": true
      },
      "concurrency_and_buffer_size": {
        "concurrency": 5000,
        "buffer_size": 10000
      },
      "custom_provider_config": {
        "base_provider_type": "openai",
        "allowed_requests": { "chat_completion": true, "chat_completion_stream": true }
      }
    }
  },
  "governance": {
    "virtual_keys": [
      {
        "id": "vk-bench",
        "name": "bench",
        "value": "sk-bf-bench-0001",
        "is_active": true,
        "provider_configs": [
          { "provider": "mock", "weight": 1.0, "allowed_models": ["*"], "key_ids": ["*"] }
        ]
      }
    ]
  },
  "config_store": {
    "enabled": true,
    "type": "sqlite",
    "config": { "path": "/app/data/config.db" }
  },
  "logs_store": {
    "enabled": true,
    "type": "sqlite",
    "config": { "path": "/app/data/logs.db" }
  }
}

For the logging-off runs, the same file has enable_logging and logs_store.enabled set to false. The data directory is wiped on every start so an old config.db cannot carry settings over. Start Bifrost pinned to cores 0-2 on the host network, which is how both gateways ran so neither paid for Docker’s NAT:

sudo rm -rf ~/bf-data; mkdir -p ~/bf-data && cp ~/harness/bifrost/config.json ~/bf-data/config.json
# https://github.com/maximhq/bifrost/releases/tag/transports/v2.4.0
sudo docker run -d --name bifrost --cpuset-cpus 0-2 --network host -e BIFROST_SETUP_TOKEN=lab-setup-token -v ~/bf-data:/app/data maximhq/bifrost:v2.4.0
until curl -sf localhost:8080/health >/dev/null; do sleep 0.2; done

LiteLLM needs a config and a Compose file. The config carries the production settings from the rig section:

vim ~/harness/litellm/config.yaml

Add the model, router and general settings:

model_list:
  - model_name: mock/gpt-4o-mini
    litellm_params:
      model: openai/gpt-4o-mini
      api_base: http://192.168.1.118:8000/v1
      api_key: mock-key-a

litellm_settings:
  set_verbose: false
  json_logs: true
  request_timeout: 600

router_settings:
  routing_strategy: simple-shuffle
  redis_host: 127.0.0.1
  redis_port: 6379

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL
  proxy_batch_write_at: 60
  database_connection_pool_limit: 10
  disable_error_logs: true
  use_redis_transaction_buffer: true

The spend-logs-off variant adds disable_spend_logs: true under general_settings. Now the Compose file, which pins the proxy to cores 0-2 and the two data services to core 3:

vim ~/harness/litellm/compose.yaml

Paste the three services:

services:
  postgres:
    image: postgres:16-alpine
    cpuset: "3"
    network_mode: host
    environment:
      POSTGRES_USER: litellm
      POSTGRES_PASSWORD: litellm-lab-pass
      POSTGRES_DB: litellm
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U litellm"]
      interval: 2s
      retries: 30

  redis:
    image: redis:7-alpine
    cpuset: "3"
    network_mode: host

  litellm:
    image: ghcr.io/berriai/litellm-database:v1.106.6
    cpuset: "0-2"
    network_mode: host
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_started
    environment:
      LITELLM_MASTER_KEY: sk-lab-master-0001
      LITELLM_SALT_KEY: sk-lab-salt-0001
      DATABASE_URL: postgresql://litellm:[email protected]:5432/litellm
      REDIS_HOST: 127.0.0.1
      REDIS_PORT: "6379"
      LITELLM_LOG: ERROR
      LITELLM_MODE: PRODUCTION
    volumes:
      - ./${LITELLM_CONFIG:-config.yaml}:/app/config.yaml:ro
    command: ["--config", "/app/config.yaml", "--port", "4000", "--num_workers", "3"]

Stop Bifrost first so only one gateway holds the cores, then bring the stack up and generate a virtual key for the load test. The first start against an empty database is the slow one, so the loop waits up to five minutes:

sudo docker rm -f bifrost
cd ~/harness/litellm && sudo docker compose up -d
for i in $(seq 1 600); do curl -sf localhost:4000/health/liveliness >/dev/null && break; sleep 0.5; done
KEY=$(curl -s -X POST localhost:4000/key/generate -H "Authorization: Bearer sk-lab-master-0001" -H "Content-Type: application/json" -d '{"key_alias":"bench"}' | jq -r .key); echo "$KEY"

The ladder script runs on the load box. It writes one Vegeta JSON report per rate into ~/results and prints a summary line, and it pins Vegeta to cores 2 and 3 so it never competes with the mock:

vim ~/ladder.sh

Add the script:

#!/bin/bash
# Run on the load-generator box. usage: ladder.sh <label> <url> <auth-header> <model> <rates...>
# Writes results/<label>-<rate>.json (vegeta report) and prints one summary line per rate.
set -u
label=$1; url=$2; auth=$3; model=$4; shift 4
mkdir -p ~/results
body=$(printf '{"model":"%s","messages":[{"role":"user","content":"Summarise the benefits of an LLM gateway in one line."}]}' "$model" | base64 -w0)
for rate in "$@"; do
  printf '{"method":"POST","url":"%s","body":"%s","header":{"Content-Type":["application/json"],"Authorization":["%s"]}}\n' \
    "$url" "$body" "$auth" > /tmp/target.json
  taskset -c 2,3 ~/bin/vegeta attack -format=json -targets=/tmp/target.json -rate="${rate}/s" -duration=30s -timeout=30s \
    | ~/bin/vegeta report -type=json > ~/results/"$label-$rate".json
  jq -r --arg l "$label" --arg r "$rate" '"\($l) rate=\($r) ok=\((.success*100)|floor)% thr=\(.throughput|floor) p50=\(.latencies["50th"]/1e6|.*100|floor/100)ms p95=\(.latencies["95th"]/1e6|.*100|floor/100)ms p99=\(.latencies["99th"]/1e6|.*100|floor/100)ms codes=\(.status_codes|tostring)"' ~/results/"$label-$rate".json
  sleep 10
done

Make it executable and run the baseline straight at the mock first. If the mock alone cannot hold a flat p50 at your top rate, the load box is the bottleneck and the gateway numbers mean nothing:

chmod +x ~/ladder.sh
~/ladder.sh direct http://localhost:8000/v1/chat/completions 'Bearer mock-key-a' gpt-4o-mini 100 250 500 1000 2000 3000 5000

Then run the same rates through each gateway. On the load box, set KEY to the LiteLLM key the gateway box printed; the Bifrost key is the one declared in its config:

~/ladder.sh bifrost http://192.168.1.150:8080/v1/chat/completions 'Bearer sk-bf-bench-0001' mock/gpt-4o-mini 100 250 500 1000 2000 3000 5000
~/ladder.sh litellm-tuned http://192.168.1.150:4000/v1/chat/completions "Bearer $KEY" mock/gpt-4o-mini 100 250 500 1000 2000 3000 5000

The streaming tests in Step 2 use two small Python scripts that need nothing beyond the standard library. The first times sequential streams:

vim ~/ttft.py

Paste the sequential TTFT script.

#!/usr/bin/env python3
"""Streaming TTFT through an OpenAI-compatible endpoint.
usage: ttft.py <url> <auth-header> <model> <runs>"""
import json, statistics, sys, time, urllib.request

url, auth, model, runs = sys.argv[1], sys.argv[2], sys.argv[3], int(sys.argv[4])
ttfts, totals = [], []
for i in range(runs + 1):
    body = json.dumps({
        "model": model, "stream": True, "max_tokens": 64, "temperature": 0,
        "messages": [{"role": "user", "content": f"Run {i}: list three uses of an LLM gateway."}],
    }).encode()
    req = urllib.request.Request(url, data=body, method="POST",
                                 headers={"Content-Type": "application/json", "Authorization": auth})
    t0 = time.perf_counter(); first = None
    with urllib.request.urlopen(req, timeout=120) as r:
        for raw in r:
            line = raw.decode().strip()
            if not line.startswith("data:") or line == "data: [DONE]":
                continue
            chunk = json.loads(line[5:])
            delta = (chunk.get("choices") or [{}])[0].get("delta", {}).get("content")
            if delta and first is None:
                first = time.perf_counter() - t0
    total = time.perf_counter() - t0
    if i == 0:
        continue  # warm-up run, model load
    ttfts.append(first * 1000); totals.append(total * 1000)
print(f"runs={runs} ttft_p50={statistics.median(ttfts):.1f}ms ttft_min={min(ttfts):.1f}ms "
      f"ttft_max={max(ttfts):.1f}ms total_p50={statistics.median(totals):.1f}ms")

Both Step 2 tests need Ollama and the gateways’ Ollama configs, which the concurrent block further down starts and describes, so set those up first. The sequential numbers came from 20 runs per target, with the label in front of each line added by sed:

python3 ~/ttft.py http://localhost:11434/v1/chat/completions "Bearer x" qwen2.5:0.5b 20 | sed 's/^/direct /'
python3 ~/ttft.py http://192.168.1.150:8080/v1/chat/completions "Bearer sk-bf-bench-0001" ollama/qwen2.5:0.5b 20 | sed 's/^/bifrost /'
python3 ~/ttft.py http://192.168.1.150:4000/v1/chat/completions "Bearer $KEY" qwen2.5:0.5b 20 | sed 's/^/litellm /'

The second script opens N streams at once, repeats that for a number of rounds and writes the raw timings to /tmp:

vim ~/ttft_concurrent.py

Paste the concurrent script.

#!/usr/bin/env python3
"""Concurrent streaming TTFT through an OpenAI-compatible endpoint.
usage: ttft_concurrent.py <label> <url> <auth-header> <model> <streams> [rounds]
Opens <streams> simultaneous streaming chats, <rounds> times, and reports TTFT and total time."""
import json, math, statistics, sys, threading, time, urllib.request

label, url, auth, model, streams = sys.argv[1:6]
streams = int(streams); rounds = int(sys.argv[6]) if len(sys.argv) > 6 else 2
ttfts, totals, errors = [], [], 0
lock = threading.Lock()


def one(i):
    global errors
    body = json.dumps({
        "model": model, "stream": True, "max_tokens": 32, "temperature": 0,
        "messages": [{"role": "user", "content": f"Stream {i}: name two uses of an LLM gateway."}],
    }).encode()
    req = urllib.request.Request(url, data=body, method="POST",
                                 headers={"Content-Type": "application/json", "Authorization": auth})
    t0 = time.perf_counter(); first = None
    try:
        with urllib.request.urlopen(req, timeout=300) as r:
            for raw in r:
                line = raw.decode().strip()
                if not line.startswith("data:") or line == "data: [DONE]":
                    continue
                delta = (json.loads(line[5:]).get("choices") or [{}])[0].get("delta", {}).get("content")
                if delta and first is None:
                    first = time.perf_counter() - t0
    except Exception:
        with lock:
            errors += 1
        return
    with lock:
        ttfts.append(first * 1000); totals.append((time.perf_counter() - t0) * 1000)


for rnd in range(rounds):
    threads = [threading.Thread(target=one, args=(rnd * streams + i,)) for i in range(streams)]
    [t.start() for t in threads]; [t.join() for t in threads]


def p95(v):
    return sorted(v)[math.ceil(len(v) * 0.95) - 1]  # nearest-rank


with open(f"/tmp/ttft-{label}-{streams}.txt", "w") as f:
    f.write("\n".join(f"{a:.1f} {b:.1f}" for a, b in zip(ttfts, totals)) + "\n")
print(f"{label} streams={streams} n={len(ttfts)} errors={errors} ttft_p50={statistics.median(ttfts):.0f}ms "
      f"ttft_p95={p95(ttfts):.0f}ms total_p50={statistics.median(totals):.0f}ms total_p95={p95(totals):.0f}ms")

Run Ollama on the load box with eight parallel slots, pull the model, then run each concurrency level direct and through the gateway. The Bifrost run below assumes it was started with an ollama provider (an ollama_key_config URL plus allow_private_network), and the LiteLLM run with an ollama_chat/qwen2.5:0.5b model entry:

# https://github.com/ollama/ollama/releases/tag/v0.12.6
sudo docker run -d --name ollama --network host -e OLLAMA_NUM_PARALLEL=8 -v ollama:/root/.ollama ollama/ollama:0.12.6
sudo docker exec ollama ollama pull qwen2.5:0.5b
python3 ~/ttft_concurrent.py direct http://127.0.0.1:11434/v1/chat/completions 'Bearer x' qwen2.5:0.5b 8 3
python3 ~/ttft_concurrent.py bifrost http://192.168.1.150:8080/v1/chat/completions 'Bearer sk-bf-bench-0001' ollama/qwen2.5:0.5b 8 3
python3 ~/ttft_concurrent.py litellm http://192.168.1.150:4000/v1/chat/completions "Bearer $KEY" qwen2.5:0.5b 8 3

Repeat the three lines with 16 and 32 in place of 8. The LiteLLM line assumes a model_list entry named qwen2.5:0.5b with model: ollama_chat/qwen2.5:0.5b and api_base: http://192.168.1.118:11434, and $KEY is a virtual key from that stack.

For Step 3, the failover script fires 100 req/s for 30 s and, in kill mode, starts a second mock on port 8001 and kills it 10 s in:

vim ~/failover.sh

Paste the failover script.

#!/bin/bash
# Run on the load-generator box. usage: failover.sh <label> <url> <auth-header> <model> <kill|nokill>
# 100 req/s for 30 s. With "kill", the second mock backend on :8001 is killed at t=10 s.
set -u
label=$1; url=$2; auth=$3; model=$4; mode=$5
mkdir -p ~/results
if [ "$mode" = kill ]; then
  pgrep -f 'mocker -host 0.0.0.0 -port 8001' >/dev/null || \
    (nohup taskset -c 0,1 ~/bin/mocker -host 0.0.0.0 -port 8001 -latency 60 >/dev/null 2>&1 < /dev/null &)
  sleep 2
  (sleep 10; pkill -f 'mocker -host 0.0.0.0 -port 8001'; date +%s.%N > ~/results/"$label"-killed-at) &
fi
body=$(printf '{"model":"%s","messages":[{"role":"user","content":"Summarise the benefits of an LLM gateway in one line."}]}' "$model" | base64 -w0)
printf '{"method":"POST","url":"%s","body":"%s","header":{"Content-Type":["application/json"],"Authorization":["%s"]}}\n' \
  "$url" "$body" "$auth" > /tmp/target-fo.json
taskset -c 2,3 ~/bin/vegeta attack -format=json -targets=/tmp/target-fo.json -rate=100/s -duration=30s -timeout=30s > ~/results/"$label".bin
~/bin/vegeta report -type=json < ~/results/"$label".bin > ~/results/"$label".json
~/bin/vegeta encode --to json < ~/results/"$label".bin > ~/results/"$label".jsonl
jq -r --arg l "$label" '"\($l) ok=\(.success*100|.*10|floor/10)% p50=\(.latencies["50th"]/1e6|.*10|floor/10)ms p99=\(.latencies["99th"]/1e6|.*10|floor/10)ms max=\(.latencies.max/1e6|floor)ms codes=\(.status_codes|tostring)"' ~/results/"$label".json

The 429 test needs the mock restarted with one key that always gets rate limited. Stop the running mock first, or the new one cannot bind port 8000 and the test quietly runs against a mock that never returns 429. On the load box:

pkill -x mocker; sleep 1
nohup taskset -c 0,1 ~/bin/mocker -host 0.0.0.0 -port 8000 -latency 60 -rate-limited-keys "Bearer mock-key-429" > ~/mocker.log 2>&1 < /dev/null &

On the gateway box, LiteLLM lists two deployments under each test model and adds the router retry settings:

vim ~/harness/litellm/config-failover.yaml

Paste the LiteLLM failover config.

model_list:
  # one key always gets 429 from the mock, the other succeeds
  - model_name: rl/gpt-4o-mini
    litellm_params:
      model: openai/gpt-4o-mini
      api_base: http://192.168.1.118:8000/v1
      api_key: mock-key-429
  - model_name: rl/gpt-4o-mini
    litellm_params:
      model: openai/gpt-4o-mini
      api_base: http://192.168.1.118:8000/v1
      api_key: mock-key-a
  # two backends; the one on :8001 is killed mid-run
  - model_name: ha/gpt-4o-mini
    litellm_params:
      model: openai/gpt-4o-mini
      api_base: http://192.168.1.118:8000/v1
      api_key: mock-key-a
  - model_name: ha/gpt-4o-mini
    litellm_params:
      model: openai/gpt-4o-mini
      api_base: http://192.168.1.118:8001/v1
      api_key: mock-key-a

litellm_settings:
  set_verbose: false
  json_logs: true
  request_timeout: 600

router_settings:
  routing_strategy: simple-shuffle
  redis_host: 127.0.0.1
  redis_port: 6379
  num_retries: 2
  allowed_fails: 1
  cooldown_time: 60

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL
  proxy_batch_write_at: 60
  database_connection_pool_limit: 10
  disable_error_logs: true
  use_redis_transaction_buffer: true

Bifrost gets three providers (the two-key rl provider and the mock/mock2 pair) and two virtual keys, one per test:

vim ~/harness/bifrost/config-failover.json

Paste the Bifrost failover config.

{
  "$schema": "https://www.getbifrost.ai/schema",
  "client": {
    "initial_pool_size": 5000,
    "drop_excess_requests": false,
    "enable_logging": true
  },
  "providers": {
    "rl": {
      "keys": [
        { "name": "key-429", "value": "mock-key-429", "models": ["gpt-4o-mini"], "weight": 1.0 },
        { "name": "key-ok", "value": "mock-key-a", "models": ["gpt-4o-mini"], "weight": 1.0 }
      ],
      "network_config": { "base_url": "http://192.168.1.118:8000", "allow_private_network": true, "max_retries": 2 },
      "custom_provider_config": { "base_provider_type": "openai", "allowed_requests": { "chat_completion": true, "chat_completion_stream": true } }
    },
    "mock": {
      "keys": [ { "name": "a", "value": "mock-key-a", "models": ["gpt-4o-mini"], "weight": 1.0 } ],
      "network_config": { "base_url": "http://192.168.1.118:8000", "allow_private_network": true },
      "custom_provider_config": { "base_provider_type": "openai", "allowed_requests": { "chat_completion": true, "chat_completion_stream": true } }
    },
    "mock2": {
      "keys": [ { "name": "b", "value": "mock-key-a", "models": ["gpt-4o-mini"], "weight": 1.0 } ],
      "network_config": { "base_url": "http://192.168.1.118:8001", "allow_private_network": true },
      "custom_provider_config": { "base_provider_type": "openai", "allowed_requests": { "chat_completion": true, "chat_completion_stream": true } }
    }
  },
  "governance": {
    "virtual_keys": [
      {
        "id": "vk-rl", "name": "rl", "value": "sk-bf-rl-0001", "is_active": true,
        "provider_configs": [ { "provider": "rl", "weight": 1.0, "allowed_models": ["*"], "key_ids": ["*"] } ]
      },
      {
        "id": "vk-ha", "name": "ha", "value": "sk-bf-ha-0001", "is_active": true,
        "provider_configs": [
          { "provider": "mock", "weight": 0.5, "allowed_models": ["*"], "key_ids": ["*"] },
          { "provider": "mock2", "weight": 0.5, "allowed_models": ["*"], "key_ids": ["*"] }
        ]
      }
    ]
  },
  "config_store": { "enabled": true, "type": "sqlite", "config": { "path": "/app/data/config.db" } },
  "logs_store": { "enabled": true, "type": "sqlite", "config": { "path": "/app/data/logs.db" } }
}

Start one gateway at a time with its failover config. The Compose file reads LITELLM_CONFIG, which has to sit after sudo or sudo drops it:

cd ~/harness/litellm && sudo LITELLM_CONFIG=config-failover.yaml docker compose up -d

Generate a key as before and run the two LiteLLM lines below from the load box. Then stop the LiteLLM stack and start Bifrost with its failover config:

cd ~/harness/litellm && sudo docker compose down
sudo docker rm -f bifrost
sudo rm -rf ~/bf-data; mkdir -p ~/bf-data && cp ~/harness/bifrost/config-failover.json ~/bf-data/config.json
sudo docker run -d --name bifrost --cpuset-cpus 0-2 --network host -e BIFROST_SETUP_TOKEN=lab-setup-token -v ~/bf-data:/app/data maximhq/bifrost:v2.4.0
until curl -sf localhost:8080/health >/dev/null; do sleep 0.2; done

These are the four failover runs, LiteLLM first, then Bifrost:

~/failover.sh litellm-429 http://192.168.1.150:4000/v1/chat/completions "Bearer $KEY" rl/gpt-4o-mini nokill
~/failover.sh litellm-kill http://192.168.1.150:4000/v1/chat/completions "Bearer $KEY" ha/gpt-4o-mini kill
~/failover.sh bifrost-429 http://192.168.1.150:8080/v1/chat/completions 'Bearer sk-bf-rl-0001' rl/gpt-4o-mini nokill
~/failover.sh bifrost-kill http://192.168.1.150:8080/v1/chat/completions 'Bearer sk-bf-ha-0001' gpt-4o-mini kill

For Step 4, the MCP server is a FastMCP script with two tools, run from a virtual environment on the load box:

python3 -m venv ~/mcpvenv && ~/mcpvenv/bin/pip install -q "mcp==1.30.0"
vim ~/lab_tools.py

Paste the server.

from mcp.server.fastmcp import FastMCP

mcp = FastMCP("lab-tools", host="0.0.0.0", port=9000)


@mcp.tool()
def disk_free(path: str = "/") -> str:
    """Return free space on a path in GiB."""
    import shutil
    total, used, free = shutil.disk_usage(path)
    return f"{path}: {free / 2**30:.1f} GiB free of {total / 2**30:.1f} GiB"


@mcp.tool()
def add(a: int, b: int) -> int:
    """Add two integers."""
    return a + b


if __name__ == "__main__":
    mcp.run(transport="streamable-http")

Each gateway then points at it. LiteLLM takes an mcp_servers block in its config.yaml:

mcp_servers:
  lab_tools:
    url: http://192.168.1.118:9000/mcp
    transport: http

Bifrost takes an mcp.client_configs entry in config.json, and the virtual key needs "mcp_configs": [ { "mcp_client_name": "lab_tools", "tools_to_execute": ["*"] } ] or it lists no tools:

  "mcp": {
    "client_configs": [
      {
        "name": "lab_tools",
        "connection_type": "http",
        "connection_string": "http://192.168.1.118:9000/mcp",
        "tools_to_execute": ["*"]
      }
    ]
  },

Last comes the probe, a real MCP client that initialises a session, lists the tools and calls add:

vim ~/mcp_probe.py

Paste the probe.

"""Connect to a gateway's MCP endpoint as a real MCP client, list tools, call one.
usage: mcp_probe.py <url> <header-name> <header-value>"""
import asyncio, sys
from mcp import ClientSession
from mcp.client.streamable_http import streamablehttp_client


async def main(url, hname, hval):
    async with streamablehttp_client(url, headers={hname: hval}) as (r, w, _):
        async with ClientSession(r, w) as s:
            init = await s.initialize()
            print("server:", init.serverInfo.name, init.serverInfo.version)
            tools = await s.list_tools()
            print("tools:", ", ".join(t.name for t in tools.tools))
            add = next(t.name for t in tools.tools if t.name.endswith("add"))
            res = await s.call_tool(add, {"a": 2, "b": 3})
            print(f"call {add}(2, 3) ->", res.content[0].text)

asyncio.run(main(*sys.argv[1:4]))

In the lab these blocks lived in their own files, config-mcp.yaml and config-mcp.json (the Ollama configs plus the MCP blocks), and each gateway was started fresh with them: sudo LITELLM_CONFIG=config-mcp.yaml docker compose up -d for LiteLLM, and the remove, copy and run block above (with config-mcp.json) for Bifrost. Editing a mounted config is not enough on its own, because neither gateway reloads it without a restart. Start the MCP server on the load box, then probe each gateway with the header it expects. With mcp 1.30.0 the probe may print a DeprecationWarning for streamablehttp_client; the call still completes:

nohup ~/mcpvenv/bin/python ~/lab_tools.py > ~/mcp.log 2>&1 < /dev/null &
~/mcpvenv/bin/python ~/mcp_probe.py http://192.168.1.150:4000/mcp x-litellm-api-key "Bearer $SCOPED_KEY"
~/mcpvenv/bin/python ~/mcp_probe.py http://192.168.1.150:8080/mcp Authorization 'Bearer sk-bf-bench-0001'

$SCOPED_KEY is a LiteLLM key created with "object_permission": {"mcp_servers": ["lab_tools"]} in the /key/generate body; a key without that grant gets HTTP 403.

Running the LiteLLM side with more workers is the obvious next experiment. On a box with 16 cores and 16 workers, the per-core rate measured here predicts roughly 560 to 880 req/s, still below what Bifrost served on three. On Proxmox, set the VM CPU type to host before you start, or the LiteLLM image will not run at all.

Where the vendor claims held up and where they did not

Every LiteLLM vs Bifrost comparison we found in November 2026 was published by Maxim, the company behind Bifrost, so each of its claims was checked against what this rig measured:

  • “LiteLLM fails at 500 RPS” (held up): the Maxim post reports 88.78% success at 500 req/s on a 2 vCPU t3.medium, against a real OpenAI key rather than a mock. Tuned LiteLLM on three cores here managed 45% at 500 req/s, and the single-worker default only 77% at 100 req/s.
  • “11 µs overhead at 5,000 RPS” (not reproduced): that figure is Bifrost’s internal processing time, which its docs say excludes JSON handling and the HTTP call. End to end through a second VM, Bifrost added 2.2 to 3.9 ms and topped out at 1,995 req/s on three cores of an older CPU.
  • “LiteLLM has no MCP gateway” (wrong): the open source LiteLLM image served /mcp to a real MCP client, listed tools and executed one, with per-key scoping.
  • “Bifrost uses far less memory” (held up at idle, not under overload): 128 to 319 MiB against 2.1 GiB at idle, but 6.0 GiB against 2.7 GiB while queuing past its ceiling.
  • “Docker size 80 MB against over 700 MB” (direction held, numbers did not): the Bifrost image is 104 MB compressed and 412 MB on disk, and the LiteLLM database image 403 MB compressed and 1.72 GB on disk.

LiteLLM’s own benchmark page deserves the same scrutiny. It reports 1,170 req/s across four instances (4 vCPU, 8 GB and 4 workers each) with 2 ms of median overhead, about 290 req/s per instance. That overhead is read from the x-litellm-overhead-duration-ms response header, so it is measured inside the proxy, the way the Bifrost 11 µs figure is. End to end on this CPU we measured roughly 35 to 55 req/s per core and 30 to 55 ms of median overhead. The honest summary is that Bifrost is an order of magnitude cheaper to run per request, and LiteLLM gives more of its governance features away. Pick on the constraint you actually have.

Keep reading

Ollama Models Cheat Sheet: Best Models by VRAM, Tested (PDF) AI Ollama Models Cheat Sheet: Best Models by VRAM, Tested (PDF) Open Source LLM Comparison Table (2026) AI Open Source LLM Comparison Table (2026) Claude Code Cheat Sheet: Commands, Shortcuts and Flags (PDF) AI Claude Code Cheat Sheet: Commands, Shortcuts and Flags (PDF) Dual GPU LLM Build: PCIe x8/x8 vs x16/x4, PSU and Cooling AI Dual GPU LLM Build: PCIe x8/x8 vs x16/x4, PSU and Cooling MCP vs RAG vs Agent Skills: When to Use Each AI MCP vs RAG vs Agent Skills: When to Use Each Install Qdrant on Ubuntu 26.04 / 24.04 / 22.04 AI Install Qdrant on Ubuntu 26.04 / 24.04 / 22.04

Leave a Comment

Press ESC to close