AI

LiteLLM Proxy: Self-Host an LLM Gateway for vLLM and Ollama

You have vLLM serving one model and Ollama serving two more, and every app on the team talks to them with a different URL and no key at all. A LiteLLM proxy puts one OpenAI-compatible endpoint in front of both, so each app gets its own virtual key, a budget, a rate limit and a fallback path, while the backends stay where they are. The request goes from your app to port 4000, LiteLLM checks the key and the limits, then picks a backend.

Original content from computingforgeeks.com - post 183794

Most LiteLLM proxy guides stop once docker compose up prints a healthy container. This one wires the whole gateway against real vLLM and Ollama backends and sets each feature up in config.yaml and again in the Admin UI: models, keys, teams, budgets, rate limits, fallbacks, Redis caching, Presidio PII masking and MCP. Only Prometheus has no UI path. Each feature is labelled open source or Enterprise. We confirmed every step in November 2026 with LiteLLM v1.106.6 on Ubuntu 24.04 lab VMs, so each output and screenshot comes from those boxes.

Check what the proxy does and what stays Enterprise

Almost everything a small platform team needs from an LLM gateway is in the MIT-licensed core. The repository’s LICENSE file puts only the enterprise/ directory under a separate licence, and the LiteLLM Enterprise page lists what that licence unlocks. Both editions ship as the same image with the same version number, so a licence key turns features on rather than swapping containers. The table maps the features in this guide, and the last column says whether we ran it in the lab or only read it in the docs:

FeatureTierChecked
Virtual keys, teams, key and team budgets, RPM/TPM limitsOpen sourceLab
Per-model budget on one key (model_max_budget), key rotationEnterpriseDocs
Load balancing, retries, cooldowns, fallbacksOpen sourceLab
Redis response cacheOpen sourceLab
Presidio PII masking guardrailOpen sourceLab
Lakera, Aporia, LLM Guard, secret hiding, guardrails per key or teamEnterpriseDocs
MCP gateway (mcp_servers)Open sourceLab
Prometheus metricsOpen sourceLab
Admin UIOpen sourceLab
SSO for the Admin UIFree up to 5 users, then EnterpriseDocs
Audit logs, JWT auth, organisations and RBAC, IP allow lists, SCIMEnterpriseDocs

Prometheus is worth calling out because older posts still list it as a paid feature. In the current docs only the managed-batch metric group carries an Enterprise banner, and the metrics below came out of an image with no licence key set. The SSO line matters more than it looks: the free tier covers five admin users, so a team that wants every engineer signing in to the UI through Okta or Entra will hit the licence wall before it hits any technical limit. If you are still choosing a gateway, our LiteLLM vs Bifrost comparison puts the two side by side, feature by feature, and the Bifrost gateway guide covers installing the Go alternative if its lower overhead matters more to you than LiteLLM’s provider count. The comparison’s benchmark numbers come from a separate rig, so don’t compare them with the CPU lab timings in this guide.

Each section below adds one piece of the request path in this diagram, so it doubles as a map of the guide.

LiteLLM proxy architecture diagram: clients call port 4000, the proxy checks keys in PostgreSQL, masks PII with Presidio, looks up Redis, then routes to vLLM or falls back to Ollama

Only the proxy is exposed to clients. PostgreSQL holds keys, teams and spend, Redis holds cached responses and the router’s shared counters, and Presidio sits beside the proxy so masking happens before a prompt ever reaches a model.

Prepare the vLLM and Ollama backends

LiteLLM does not run models, so the two backends have to answer on their own before the gateway is worth starting. If you already run them, skip to the next section and note each one’s base URL. Our lab had no GPU budget, so we ran vLLM’s CPU image with a small model. The proxy treats a CPU vLLM and a GPU vLLM identically (same OpenAI API, same /v1/models), so every gateway behaviour below carries over, and only the latency numbers are CPU numbers. For a GPU install, follow the vLLM production guide and the Ollama setup on Ubuntu, then come back here. Our lab put everything on one host, which is fine for learning the gateway and wrong for production. The overhead numbers in the tuning section show the proxy slowing down when it competes with the models for CPU, so treat this list as a floor and give the gateway its own small VM once real clients depend on it.

  • Docker Engine with the Compose plugin (the Docker cheat sheet covers the commands used here)
  • 4 vCPU and 16 GB RAM for the gateway, both backends and Presidio on one box; the gateway alone needs far less, see the production section
  • Backends reachable from the gateway: vLLM on 10.0.1.50:8000 and Ollama on 10.0.1.50:11434 in our lab, so replace that address with your GPU host’s
  • Port 4000 (proxy) open to your clients, and 9090 (Prometheus, optional) open to admins only, since Prometheus has no authentication (Docker-published ports bypass ufw and firewalld, so restrict 9090 in a cloud security group or the DOCKER-USER chain)

Create the backend directory and its Compose file. Both services publish their ports on the host, because the gateway will reach them over the network exactly as it would reach a separate GPU server.

mkdir -p ~/backends && vim ~/backends/compose.yaml

We ran exactly this file in the lab. On an AVX2-only CPU like ours, --dtype float32 is the line that matters, because bfloat16 left the engine hanging for 19 minutes without an error. Drop it only on AVX-512 hardware, and see the troubleshooting section at the end for the log that gives it away.

services:
  vllm:
    image: vllm/vllm-openai-cpu:v0.33.0-x86_64 # https://hub.docker.com/r/vllm/vllm-openai-cpu/tags
    ports:
      - "8000:8000"
    command:
      - --model
      - Qwen/Qwen2.5-0.5B-Instruct
      - --served-model-name
      - qwen2.5-0.5b-instruct
      - --max-model-len
      - "4096"
      - --dtype
      - float32
    environment:
      VLLM_CPU_KVCACHE_SPACE: "2"
    volumes:
      - hf-cache:/root/.cache/huggingface
    ipc: host
    security_opt:
      - seccomp=unconfined
    cap_add:
      - SYS_NICE
    restart: unless-stopped
  ollama:
    image: ollama/ollama:0.40.14 # https://github.com/ollama/ollama/releases
    ports:
      - "11434:11434"
    environment:
      OLLAMA_KEEP_ALIVE: 24h
    volumes:
      - ollama:/root/.ollama
    restart: unless-stopped
volumes:
  hf-cache:
  ollama:

Start both backends and pull the two Ollama models the gateway will route to. The vLLM container downloads Qwen2.5-0.5B-Instruct from Hugging Face on its first start, so expect the first check below to fail for a few minutes.

cd ~/backends
docker compose up -d
docker compose exec ollama ollama pull qwen2.5:0.5b
docker compose exec ollama ollama pull llama3.2:1b

The vLLM CPU engine compiles its graph on first start, which took 329 seconds on our 4 vCPU VM, and restarts reuse the compiled cache. Once /v1/models answers, the backend is ready. Note max_model_len while you are here, because a fallback model with a smaller window truncates long prompts without telling anyone, as the troubleshooting section shows.

curl -s http://10.0.1.50:8000/v1/models | jq '.data[] | {id, max_model_len}'

Copy the id exactly into the gateway config later, since it is the served name LiteLLM asks vLLM for. A typo there comes back later as a model-not-found error from vLLM, which is easy to mistake for a gateway problem. Paste it rather than retyping it.

{
  "id": "qwen2.5-0.5b-instruct",
  "max_model_len": 4096
}

Check the Ollama side the same way. Ollama answers by tag, so qwen2.5:0.5b with its colon is the exact string the gateway config needs.

docker compose exec ollama ollama list

Both models are present, at 397 MB and 1.3 GB on disk. With OLLAMA_KEEP_ALIVE at 24h, a model stays in memory after its first request, so only that first call through the gateway pays the load time.

NAME            ID              SIZE      MODIFIED
llama3.2:1b     baf6a787fdff    1.3 GB    28 minutes ago
qwen2.5:0.5b    a8b0c5157701    397 MB    30 minutes ago    

Called directly, vLLM took 8.6 seconds for a 40-token answer and Ollama 3.7 seconds for 118 tokens on this CPU, which is the baseline the gateway’s own overhead gets measured against later. If you are choosing between the two engines for real traffic, our Ollama vs vLLM vs llama.cpp benchmark has the GPU numbers.

Deploy the LiteLLM proxy with Docker Compose

The gateway is three containers: the proxy, PostgreSQL for keys and spend, and Redis for the cache and the counters that several workers share. Virtual keys need the database, so a proxy without PostgreSQL can only use the master key. Start with the secrets, because two of them can’t change once data exists. The salt key encrypts any provider credentials stored in the database, and the docs warn that changing it later makes those credentials unreadable. Create the directory and write the .env file. The version line asks GitHub for the latest stable release, which skips the -dev and -rc builds LiteLLM publishes almost daily.

mkdir -p ~/litellm && cd ~/litellm
LITELLM_VERSION=$(curl -s https://api.github.com/repos/BerriAI/litellm/releases/latest | jq -r .tag_name)
echo "LITELLM_VERSION=${LITELLM_VERSION}" > .env
echo "LITELLM_MASTER_KEY=sk-$(openssl rand -hex 24)" >> .env
echo "LITELLM_SALT_KEY=sk-$(openssl rand -hex 24)" >> .env
echo "POSTGRES_PASSWORD=$(openssl rand -hex 16)" >> .env
echo "UI_USERNAME=admin" >> .env
echo "UI_PASSWORD=$(openssl rand -base64 18)" >> .env
chmod 600 .env
sed -n 's/^LITELLM_VERSION=//p' .env

Expect a plain release tag with no suffix. If it prints null, the GitHub API refused the call (its anonymous rate limit is low), so set the tag by hand from the releases page before going on.

v1.106.6

Open the Compose file. It defines three services and three named volumes, and nothing else needs installing on the host beyond Docker.

vim ~/litellm/compose.yaml

Paste the core stack. We use the ghcr.io/berriai/litellm image rather than litellm-database, because the deploy docs point Postgres-backed deployments at the core image, which bundles Prisma itself.

services:
  litellm:
    image: ghcr.io/berriai/litellm:${LITELLM_VERSION}
    command: ["--config", "/app/config.yaml", "--port", "4000", "--num_workers", "4"]
    ports:
      - "4000:4000"
    volumes:
      - ./config.yaml:/app/config.yaml:ro
    env_file: .env
    environment:
      DATABASE_URL: postgresql://litellm:${POSTGRES_PASSWORD}@db:5432/litellm
      REDIS_HOST: redis
      REDIS_PORT: "6379"
      STORE_MODEL_IN_DB: "True"
    depends_on:
      db:
        condition: service_healthy
      redis:
        condition: service_healthy
    healthcheck:
      test: ["CMD-SHELL", "python3 -c \"import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')\""]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 60s
    restart: unless-stopped

  db:
    image: postgres:16
    environment:
      POSTGRES_DB: litellm
      POSTGRES_USER: litellm
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -d litellm -U litellm"]
      interval: 5s
      timeout: 5s
      retries: 10
    restart: unless-stopped

  redis:
    image: redis:7.4-alpine
    command: ["redis-server", "--appendonly", "yes"]
    volumes:
      - redisdata:/data
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 10
    restart: unless-stopped

volumes:
  pgdata:
  redisdata:

STORE_MODEL_IN_DB is what lets the Admin UI add models and MCP servers later; without it the UI can only show what config.yaml defines. Four workers match the four vCPUs, which is the docs’ rule for a single VM, and PostgreSQL is not published on the host because only the proxy talks to it.

Point the LiteLLM proxy at vLLM and Ollama

Each backend becomes one entry in model_list: a public model_name that clients send, and litellm_params that say how to reach the backend. The prefix picks the provider adapter. The vLLM docs page recommends hosted_vllm/ for an OpenAI-compatible vLLM server, and the Ollama page recommends ollama_chat/, which calls /api/chat instead of the older generate endpoint. We also set a price per token on both, which local models don’t have, because budgets only bite when spend is above zero.

vim ~/litellm/config.yaml

Start with the two backends and the master key. Every later section adds one block to this file, so you can restart after each step and see exactly what changed.

model_list:
  - model_name: qwen-vllm
    litellm_params:
      model: hosted_vllm/qwen2.5-0.5b-instruct
      api_base: http://10.0.1.50:8000/v1
    model_info:
      input_cost_per_token: 0.00001
      output_cost_per_token: 0.00003

  - model_name: qwen-ollama
    litellm_params:
      model: ollama_chat/qwen2.5:0.5b
      api_base: http://10.0.1.50:11434
    model_info:
      input_cost_per_token: 0.00001
      output_cost_per_token: 0.00003

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

The vLLM base URL ends in /v1 and the Ollama one does not; mixing those up is the most common first-run error with this pair. Start the stack and wait for readiness, which only returns healthy once PostgreSQL is connected.

cd ~/litellm
docker compose up -d
until curl -sf localhost:4000/health/readiness >/dev/null; do sleep 5; done
curl -s localhost:4000/health/readiness | jq

On first start the proxy runs the Prisma migrations, which took a little over a minute here, so leave the container alone while readiness still fails.

{
  "status": "healthy",
  "db": "connected"
}

Load the secrets into your shell so the following commands can use the master key, then send one request through the gateway to each backend. A new shell needs this line again.

cd ~/litellm; set -a; . ./.env; set +a
curl -s -D /tmp/h.txt http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen-vllm", "messages": [{"role": "user", "content": "Name one Linux init system."}], "max_tokens": 40}' | jq '{model, content: .choices[0].message.content, usage}'
grep -iE "^x-litellm-(model-api-base|response-cost|attempted|overhead|response-duration)" /tmp/h.txt

The response headers are where the gateway explains itself. x-litellm-model-api-base names the backend that answered, the cost headers show the custom price applied to 35 input and 40 output tokens, and the retry and fallback counters are zero because nothing failed.

{
  "model": "qwen-vllm",
  "content": "One of the most commonly used Linux init systems is systemd (short for \"Systemd\"). systemd is designed to provide a centralized and automated approach to managing services on a Linux system. It uses a set",
  "usage": {
    "completion_tokens": 40,
    "prompt_tokens": 35,
    "total_tokens": 75
  }
}
x-litellm-model-api-base: http://10.0.1.50:8000/v1
x-litellm-response-cost: 0.0015500000000000002
x-litellm-response-cost-original: 0.0015500000000000002
x-litellm-response-cost-discount-amount: 0.0
x-litellm-response-cost-margin-amount: 0.0
x-litellm-response-cost-margin-percent: 0.0
x-litellm-response-cost-input: 0.00035000000000000005
x-litellm-response-cost-output: 0.0012000000000000001
x-litellm-response-cost-tool-usage: 0.0
x-litellm-response-duration-ms: 13236.500978469849
x-litellm-overhead-duration-ms: 3118.1078
x-litellm-attempted-retries: 0
x-litellm-attempted-fallbacks: 0

That 3.1-second overhead is a cold first request on a freshly restarted proxy, not the steady state; once the workers are warm the same header reads 12 to 56 ms, as the production section shows. The same request with "model": "qwen-ollama" came back from http://10.0.1.50:11434, so the gateway can now reach both sides of the wiring.

Sign in to the Admin UI and add a model there

The Admin UI lives at http://10.0.1.50:4000/ui on the same port as the API (our gateway shared the backend host, so use your gateway’s own address if they are separate). Sign in with UI_USERNAME and UI_PASSWORD from .env (if UI_PASSWORD is unset, the master key works as the password). The first thing it shows is a red banner about how you signed in, and the warning is fair.

LiteLLM Admin UI home page with the environment-credential login warning banner and request totals

Anyone holding the environment credentials signs in as a proxy admin with a shared static secret. The banner’s own fix is to create a regular admin account with its own password under Internal Users, then set disable_env_credential_login: true under general_settings. Do it in that order, because flipping the setting first leaves you with no way into the UI until you remove it again and restart the proxy.

Models defined in config.yaml appear under Models + Endpoints as “Defined in config” and can’t be edited there. A model added in the UI is stored in PostgreSQL and goes live on every worker without a restart, which is the path a team uses for experiments. To add the second Ollama model as llama-ollama, open Models + Endpoints, then Add Model.

  1. Set Provider to Ollama Chat, which maps to the ollama_chat/ prefix.
  2. Type llama3.2:1b in LiteLLM Model Name(s), the exact name Ollama lists.
  3. Change the Public Model Name to llama-ollama, the name clients will send.
LiteLLM Admin UI Add Model form with Ollama Chat provider, llama3.2:1b and public name llama-ollama
  1. 1Provider
  2. 2LiteLLM Model Name(s)
  3. 3Public Model Name

Scroll down, set API Base to http://10.0.1.50:11434, and click Add Model. Leave the model name field with the mouse, not the Enter key: in our run, Enter submitted the half-filled form and created a model with no API base, which then failed every request until we deleted it.

API Base set to the Ollama server and the Add Model button in the LiteLLM Admin UI
  1. 1API Base
  2. 2Add Model

Back on Deployed Models, the new entry sits under the config models with no “Defined in config” label. That missing label means the UI can edit or delete it, unlike the config entries above it.

LiteLLM Deployed Models list showing qwen-vllm, qwen-ollama, two qwen-small deployments and the UI-added llama-ollama

A quick call by the new public name confirms the database-stored model is live on every worker. No restart was needed, which is the main reason to add experimental models this way.

curl -s http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" \
  -d '{"model": "llama-ollama", "messages": [{"role": "user", "content": "Which company makes Llama models? One word."}], "max_tokens": 10}' | jq -r '.model, .choices[0].message.content'

Routing worked and the 1B model got the fact wrong, which is a fair reminder that the gateway checks access, not answers.

llama-ollama
Google

The two qwen-small rows in that list come from the load-balancing section further down. We’d keep anything production traffic depends on in config.yaml, under version control, and use the UI for models you are trying out.

Issue virtual keys with teams and budgets

A virtual key is what you hand to an app instead of the master key. It carries its own allowed models, budget and limits, and it can belong to a team whose own budget caps every key in it. Create the team first through the API, so the key has somewhere to live.

curl -s http://localhost:4000/team/new \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"team_alias": "platform", "models": ["qwen-vllm", "qwen-ollama"], "max_budget": 5, "budget_duration": "30d", "rpm_limit": 120}' | jq '{team_id, team_alias, models, max_budget, budget_duration, rpm_limit}'

Every key created inside this team now sits under a ceiling of $5 a month across both models and 120 requests a minute, even if the key’s own budget is higher. That is the lever for a platform team, because it caps a whole product line without touching each key.

{
  "team_id": "d9930005-1ce8-4a33-bc0e-2f12c7c70baf",
  "team_alias": "platform",
  "models": [
    "qwen-vllm",
    "qwen-ollama"
  ],
  "max_budget": 5.0,
  "budget_duration": "30d",
  "rpm_limit": 120
}

Now issue a key for a CI job inside that team. We give it one model, one cent of budget and three requests a minute, so both limits are easy to hit on purpose. The key is shown once, so the command saves it to a file.

TEAM_ID=$(curl -s "http://localhost:4000/team/list" -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -r '.[] | select(.team_alias=="platform") | .team_id')
curl -s http://localhost:4000/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d "{\"key_alias\": \"ci-bot\", \"team_id\": \"${TEAM_ID}\", \"models\": [\"qwen-ollama\"], \"max_budget\": 0.01, \"rpm_limit\": 3}" | tee /tmp/ci-key.json | jq '{key_alias, key, models, max_budget, rpm_limit, team_id}'
jq -r .key /tmp/ci-key.json > ~/ci.key

We masked the key in this output, and yours prints in full. Store it in your CI secrets straight away, because the proxy keeps only a hash of it (the long hex string in the rate-limit error below), so the full value can’t be shown again. Lose it and you issue a new one.

{
  "key_alias": "ci-bot",
  "key": "sk-me6U****",
  "models": [
    "qwen-ollama"
  ],
  "max_budget": 0.01,
  "rpm_limit": 3,
  "team_id": "d9930005-1ce8-4a33-bc0e-2f12c7c70baf"
}

The model list is enforced before any backend is called. Asking for qwen-vllm with this key fails at the gateway with a 403, even though the team itself is allowed that model.

export CI_KEY=$(cat ~/ci.key)
curl -s http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer $CI_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen-vllm", "messages": [{"role": "user", "content": "hi"}]}' | jq

Match on the type field when you alert on this, since key_model_access_denied is a fixed machine-readable value and the message is prose that can be reworded.

{
  "error": {
    "message": "The requested model 'qwen-vllm' is not available for this API key, or the model name is invalid. Check the models available to you and try again.",
    "type": "key_model_access_denied",
    "param": "model",
    "code": "403"
  }
}

The same key can be built in the UI. Open Virtual Keys, click Create New Key, and fill the three fields that decide what the key reaches.

  1. Pick platform in Team, so the team budget caps this key too.
  2. Enter ui-demo as the Key Name, the alias that shows in logs.
  3. Choose qwen-vllm in Models. Leaving it empty grants every team model.
Create New Key dialog in LiteLLM with the platform team, key name ui-demo and the qwen-vllm model
  1. 1Team
  2. 2Key Name
  3. 3Models

Expand Optional Settings for the limits. Set Max Budget (USD) to 2 and Requests per minute Limit (RPM) to 30, then click Create Key. The dialog that follows shows the key once, exactly like the API response.

Max Budget and RPM limit fields under Optional Settings of a LiteLLM virtual key
  1. 1Max Budget (USD)
  2. 2Requests per minute Limit (RPM)

Listing the keys confirms that both paths write the same record, one from curl and one from the dialog. Nothing marks where a key came from afterwards, so pick whichever path fits the workflow.

curl -s "http://localhost:4000/key/list?return_full_object=true" -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  | jq -r '.keys[] | "\(.key_alias)\tmodels=\(.models|join(","))\tmax_budget=\(.max_budget)\trpm=\(.rpm_limit)\tspend=\(.spend)"'

Spend on ci-bot is already over its cent because this listing ran after the budget test in the next section.

ui-demo	models=qwen-vllm	max_budget=2.0	rpm=30	spend=0.0
ci-bot	models=qwen-ollama	max_budget=0.01	rpm=3	spend=0.01104

Open Teams to see the same team with its budget and members, which is where you’d check who is close to the cap.

LiteLLM Teams page listing the platform team with its models, budget and rate limit

Read one rule in the docs before you design teams. A key that belongs to a team is checked against the team budget and the member’s budget inside that team, not against the owner’s personal budget.

Create a team in the Admin UI

The platform team came from the API, and the UI builds the same record. We created a second team, ml-research, on a fresh gateway this way. Open Teams, click Create Team, and fill the dialog from the top down.

  1. Type ml-research in Team Name.
  2. Pick qwen-ollama in Models.
  3. Enter 10 in Max Budget (USD).
  4. Choose monthly in Reset Budget.
  5. Enter 60 in Requests per minute Limit (RPM), then click Create Team at the bottom of the dialog.
Create Team dialog in the LiteLLM Admin UI with team name, model, max budget, monthly budget reset and RPM limit
  1. 1Team Name
  2. 2Models
  3. 3Max Budget (USD)
  4. 4Reset Budget
  5. 5Requests per minute Limit (RPM)

The Per-Model Budgets row in that dialog is marked as a premium feature, which matches the Enterprise label on per-model budgets in the table at the top. Read the team back through the API to confirm what the UI stored.

curl -s http://localhost:4000/team/list -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  | jq '.[] | select(.team_alias=="ml-research") | {team_alias, models, max_budget, budget_duration, rpm_limit, budget_reset_at}'

“monthly” became budget_duration: 30d, but the first reset falls on 1 December, the first day of the next calendar month, not 30 days after the team was created. Plan quota reviews around that date.

{
  "team_alias": "ml-research",
  "models": [
    "qwen-ollama"
  ],
  "max_budget": 10.0,
  "budget_duration": "30d",
  "rpm_limit": 60,
  "budget_reset_at": "2026-12-01T00:00:00Z"
}

A team made in the UI behaves exactly like one made with curl, so keys issued into it inherit the same budget and RPM ceiling.

Enforce rate limits and watch spend

Both limits are enforced at the gateway, so the backend never sees the rejected request. The RPM limit is the quicker one to prove: fire four requests inside a minute with a key allowed three.

for i in 1 2 3 4; do
  curl -s -o /dev/null -w "request $i: HTTP %{http_code}\n" http://localhost:4000/v1/chat/completions \
    -H "Authorization: Bearer $CI_KEY" -H "Content-Type: application/json" \
    -d '{"model": "qwen-ollama", "messages": [{"role": "user", "content": "Say ok"}], "max_tokens": 5}'
done

Only the fourth request fails, with a 429 from the proxy itself. Ollama never saw it, so a client stuck in a retry loop costs the gateway a Redis lookup and nothing more. Three a minute is only a test value, so size the real limit from a week of the key’s traffic in the logs.

Terminal showing a LiteLLM virtual key with rpm_limit 3 returning HTTP 429 on the fourth request

A well-behaved SDK needs to know when to come back, and the body of that 429 says so with an exact reset time. Pass it to your retry logic rather than retrying blindly, because every blind retry inside the window is just another rejected request.

{
  "message": "Rate limit exceeded for api_key: 3eb688cca8dcdcd207f9ceda909672b4da1c923323fab65a755febcd7a8e4b97. Limit type: requests. Current limit: 3, Remaining: 0. Limit resets at: 2026-11-03 18:59:43 UTC",
  "type": "throttling_error",
  "param": null,
  "code": "429"
}

All four workers counted against the same limit, because the counters are shared rather than per process. Wait a minute for that window to clear before the budget test, or its first request hits the same 429. For the budget, we spaced requests 21 seconds apart to stay under the RPM limit and read the key’s spend after each one.

for i in $(seq 1 9); do
  code=$(curl -s -o /tmp/r.json -w "%{http_code}" http://localhost:4000/v1/chat/completions \
    -H "Authorization: Bearer $CI_KEY" -H "Content-Type: application/json" \
    -d '{"model": "qwen-ollama", "messages": [{"role": "user", "content": "Explain DNS in two sentences."}], "max_tokens": 60}')
  spend=$(curl -s "http://localhost:4000/key/info" -H "Authorization: Bearer $CI_KEY" | jq -r .info.spend)
  echo "request $i: HTTP $code  key spend so far: \$$spend"
  [ "$code" != "200" ] && jq .error /tmp/r.json && break
  sleep 21
done

Each answer cost about $0.0016 at our made-up prices, so the seventh request found the key over its cent. Those prices only exist to make the arithmetic visible, so set real per-token figures from your GPU cost if you want budgets to mean money. A local GPU has no list price, so its hourly cost divided by the tokens it serves in an hour is a fair starting figure.

request 1: HTTP 200  key spend so far: $0.00138
request 2: HTTP 200  key spend so far: $0.0033
request 3: HTTP 200  key spend so far: $0.00456
request 4: HTTP 200  key spend so far: $0.00615
request 5: HTTP 200  key spend so far: $0.00795
request 6: HTTP 200  key spend so far: $0.00969
request 7: HTTP 422  key spend so far: $0.01104
{
  "message": "Budget has been exceeded! Key=ci-bot (sk-...9kEg) Current cost: 0.01104, Max budget: 0.01",
  "type": "budget_exceeded",
  "param": null,
  "code": "422"
}

Read the last two lines together. After request 6, /key/info still said $0.00969, yet request 7 was refused at $0.01104. The proxy enforces against its live counter, while the database copy that /key/info reads is written in batches, so it trails by a request or so. A budget is therefore a ceiling you can overshoot by one request, never a hard stop at the exact cent, and the overshoot grows with proxy_batch_write_at. Budgets also reset on their own when you set budget_duration, so a monthly key doesn’t need a cron job. Every request, refused or not, lands in Logs with its cost, duration, model and key, and Usage rolls them up by model and key.

LiteLLM Usage page with total spend, requests, tokens and spend split between qwen-ollama and qwen-vllm

That view covers the whole lab session: 86 requests, 11 of them failed on purpose, and Ollama carried 91.7% of the spend because the fallback tests pushed traffic onto it.

Prove fallbacks and load balancing with a dead backend

LiteLLM gives you two separate safety nets, and they answer different failures. Load balancing spreads one public model name over several deployments and retries a failed call on another deployment of that same name. A fallback switches to a different model name when every deployment of the first one has failed. Add both to config.yaml: a qwen-small name with one deployment on each backend, the router settings, and a fallback from qwen-vllm to qwen-ollama. Begin by appending the two qwen-small deployments to model_list.

  # one public name, two deployments: the router spreads traffic across both
  - model_name: qwen-small
    litellm_params:
      model: hosted_vllm/qwen2.5-0.5b-instruct
      api_base: http://10.0.1.50:8000/v1
    model_info:
      input_cost_per_token: 0.00001
      output_cost_per_token: 0.00003

  - model_name: qwen-small
    litellm_params:
      model: ollama_chat/qwen2.5:0.5b
      api_base: http://10.0.1.50:11434
    model_info:
      input_cost_per_token: 0.00001
      output_cost_per_token: 0.00003

Then add the router settings and the fallback map as top-level keys. Both blocks sit at the same level as model_list, not inside it, and take effect when the proxy restarts.

router_settings:
  routing_strategy: simple-shuffle
  num_retries: 2
  allowed_fails: 1
  cooldown_time: 30
  redis_host: os.environ/REDIS_HOST
  redis_port: os.environ/REDIS_PORT

litellm_settings:
  request_timeout: 120
  fallbacks: [{"qwen-vllm": ["qwen-ollama"]}]

allowed_fails: 1 with cooldown_time: 30 means a deployment that fails more than once is benched for 30 seconds, and the Redis settings make that cooldown visible to all four workers instead of each worker learning it separately. Restart the proxy with docker compose restart litellm, then send ten requests to the balanced name with a distinct prompt each, so the cache added later never answers for them.

for i in $(seq 1 10); do
  curl -s -o /dev/null -D - http://localhost:4000/v1/chat/completions \
    -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" \
    -d "{\"model\": \"qwen-small\", \"messages\": [{\"role\": \"user\", \"content\": \"Reply with the number $i\"}], \"max_tokens\": 5}" \
    | grep -i '^x-litellm-model-api-base'
done | sort | uniq -c

Both backends took traffic. Simple shuffle is random, so a 7 to 3 split over ten requests is normal; add rpm or weight to a deployment when you want the GPU box to take a fixed share.

      3 x-litellm-model-api-base: http://10.0.1.50:11434
      7 x-litellm-model-api-base: http://10.0.1.50:8000/v1

Now kill the vLLM side of the wiring and ask for qwen-vllm by name. That model has a single deployment, so retries can’t save it, and the fallback has to.

docker compose -f ~/backends/compose.yaml stop vllm
curl -s -D /tmp/h.txt http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" \
  -d '{"model": "qwen-vllm", "messages": [{"role": "user", "content": "Is the backup model answering?"}], "max_tokens": 20}' | jq -r '.model, .choices[0].message.content'
grep -iE "^x-litellm-(model-api-base|attempted-retries|attempted-fallbacks|response-duration-ms)" /tmp/h.txt

Asked for qwen-vllm, the gateway answered from Ollama and recorded one fallback. No client code changed, and nothing in the request hinted that the primary was down.

Terminal showing vLLM stopped and LiteLLM answering qwen-vllm requests from Ollama with x-litellm-attempted-fallbacks 1

The model field in the response changes to ollama_chat/qwen2.5:0.5b, so an app that logs it can tell a fallback answer from a normal one. That request took 18.7 seconds, most of it spent finding out that vLLM was gone, which is the cost of a fallback you should plan your client timeouts around. With vLLM still down, the balanced name behaved differently.

for i in $(seq 1 6); do
  curl -s -o /dev/null -w "HTTP %{http_code} %{time_total}s " -D /tmp/h$i.txt http://localhost:4000/v1/chat/completions \
    -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" \
    -d "{\"model\": \"qwen-small\", \"messages\": [{\"role\": \"user\", \"content\": \"Reply with the letter $i\"}], \"max_tokens\": 5}"
  grep -iE '^x-litellm-(model-api-base|attempted-retries)' /tmp/h$i.txt | tr -d '\r' | tr '\n' ' '; echo
done

Every request succeeded on Ollama. The first needed two retries and the second one retry, then the dead deployment was in cooldown and the router stopped trying it at all.

HTTP 200 9.686201s x-litellm-model-api-base: http://10.0.1.50:11434 x-litellm-attempted-retries: 2
HTTP 200 4.101894s x-litellm-model-api-base: http://10.0.1.50:11434 x-litellm-attempted-retries: 1
HTTP 200 10.438740s x-litellm-model-api-base: http://10.0.1.50:11434 x-litellm-attempted-retries: 0
HTTP 200 4.208327s x-litellm-model-api-base: http://10.0.1.50:11434 x-litellm-attempted-retries: 0
HTTP 200 12.026324s x-litellm-model-api-base: http://10.0.1.50:11434 x-litellm-attempted-retries: 0
HTTP 200 10.849991s x-litellm-model-api-base: http://10.0.1.50:11434 x-litellm-attempted-retries: 0

That is the reason to prefer a balanced name over a fallback for routine traffic: the balanced name stops paying the failure cost after two requests, while a fallback pays it on every request until the primary returns. Start vLLM again with docker compose -f ~/backends/compose.yaml start vllm; with the compiled cache in place it was serving again in about three minutes.

Set retries and fallbacks in the Admin UI

The routing strategy, retries, cooldown, timeout and the fallback map can also be set under Settings, Router Settings. The Redis connection for the router’s shared counters is not on that page, so it stays in the config or the environment. To prove the UI path on its own, we ran it on a second gateway whose config.yaml held only the two models, with no router block and no fallbacks. On the Loadbalancing tab:

  1. Set Allowed Fails to 1.
  2. Set Cooldown Time to 30.
  3. Set Number of Retries to 2.
  4. Set Timeout to 120.
  5. Click Save Changes.
LiteLLM Router Settings Loadbalancing tab with allowed fails, cooldown time, number of retries and timeout
  1. 1Allowed Fails
  2. 2Cooldown Time
  3. 3Number of Retries
  4. 4Timeout
  5. 5Save Changes

Then open the Fallbacks tab and click Add Fallbacks. Pick qwen-vllm in Primary Model, add qwen-ollama to the Fallback Chain, and click Save All Configurations. The chain takes up to 10 models, tried in the order listed.

Configure Model Fallbacks dialog in LiteLLM with qwen-vllm as primary model and qwen-ollama in the fallback chain
  1. 1Primary Model
  2. 2Fallback Chain
  3. 3Save All Configurations

Always read the values back after saving. Our first scripted pass filled the wrong boxes, storing 30 retries and a 2-second timeout, and two fallback requests in a row then took 235 seconds each before Ollama answered. The /router/settings endpoint shows what the router will actually use.

curl -s localhost:4000/router/settings -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  | jq -c '.fields[] | select(.field_name|test("^(num_retries|timeout|fallbacks|allowed_fails|cooldown_time)$")) | {field_name, field_value}'

After the fix, all five values match the form. Compare them every time you save, because the page accepts any number in any box and shows no warning.

{"field_name":"num_retries","field_value":2}
{"field_name":"timeout","field_value":120.0}
{"field_name":"fallbacks","field_value":[{"qwen-vllm":["qwen-ollama"]}]}
{"field_name":"allowed_fails","field_value":1}
{"field_name":"cooldown_time","field_value":30.0}

Nothing listened on port 8000 on that gateway’s host, so a qwen-vllm request could only succeed through the UI-defined fallback. It did, in 5.4 seconds, with the same headers as the config-file test.

ollama_chat/qwen2.5:0.5b
I'm sorry, but I'm Qwen, a language model created by Alibaba Cloud. I don
x-litellm-model-api-base: http://10.0.1.50:11434
x-litellm-response-duration-ms: 5378.1938552856445
x-litellm-attempted-retries: 0
x-litellm-attempted-fallbacks: 1

That output came from the same curl and grep pair used for the stopped-vLLM test above. Keep production routing in config.yaml so it lives in version control, and use the UI page to check what is live.

Inject 429s from a backend and watch the cooldown

A stopped container fails fast with a connection error. A busy upstream fails slower, with HTTP 429, and that is the case worth testing before production. We wrote a 15-line backend that answers every chat call with a 429.

vim ~/litellm/mock429.py

Only the Python standard library is needed, so any small Python image can run it. The Retry-After header mirrors what a real provider sends.

from http.server import BaseHTTPRequestHandler, HTTPServer


class TooMany(BaseHTTPRequestHandler):
    def do_POST(self):
        self.send_response(429)
        self.send_header("Content-Type", "application/json")
        self.send_header("Retry-After", "30")
        self.end_headers()
        self.wfile.write(b'{"error": {"message": "rate limited by upstream", "type": "rate_limit_error"}}')

    def log_message(self, fmt, *args):
        print("429 ->", self.path, flush=True)


HTTPServer(("0.0.0.0", 8099), TooMany).serve_forever()

Run it in a throwaway container on the backend address. Port 8099 then plays an upstream that is overloaded but still reachable. The connection succeeds every time, so the router has to learn about the failure from the response code instead of a refused connection.

docker run -d --name mock429 -p 8099:8099 -v $PWD/mock429.py:/mock429.py:ro python:3.12-alpine python -u /mock429.py

Then add it to model_list as a third qwen-small deployment and restart the proxy. The fixed id makes it easy to find in metrics.

  - model_name: qwen-small
    litellm_params:
      model: hosted_vllm/qwen2.5-0.5b-instruct
      api_base: http://10.0.1.50:8099/v1
    model_info:
      id: flaky-429

Twelve requests to qwen-small, each with a unique prompt. A repeated prompt is answered from the Redis cache, and the cache hit still reports a backend in its headers, which wrecked our first attempt at this test.

run=$(date +%s)
for i in $(seq 1 12); do
  curl -s -o /dev/null -w "$(date +%T) req $i HTTP %{http_code} " -D /tmp/h.txt http://localhost:4000/v1/chat/completions \
    -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" \
    -d "{\"model\": \"qwen-small\", \"messages\": [{\"role\": \"user\", \"content\": \"Run $run, reply with $i\"}], \"max_tokens\": 5}"
  grep -iE '^x-litellm-(model-api-base|attempted-retries)' /tmp/h.txt | tr -d '\r' | tr '\n' ' '; echo
done
echo "chat requests the 429 mock rejected:"
docker logs -t mock429 2>&1 | grep 'chat/completions' | cut -c12-19

All 12 returned 200 to the client. The mock rejected exactly two calls, six seconds apart, and each one shows up as a single retry on the request that hit it. After the second failure the flaky deployment was cooled down and got no more traffic for the rest of the run.

19:29:22 req 1 HTTP 200 x-litellm-model-api-base: http://10.0.1.50:8000/v1 x-litellm-attempted-retries: 1
19:29:22 req 2 HTTP 200 x-litellm-model-api-base: http://10.0.1.50:11434 x-litellm-attempted-retries: 0
19:29:23 req 3 HTTP 200 x-litellm-model-api-base: http://10.0.1.50:8000/v1 x-litellm-attempted-retries: 0
19:29:24 req 4 HTTP 200 x-litellm-model-api-base: http://10.0.1.50:11434 x-litellm-attempted-retries: 0
19:29:25 req 5 HTTP 200 x-litellm-model-api-base: http://10.0.1.50:8000/v1 x-litellm-attempted-retries: 0
19:29:26 req 6 HTTP 200 x-litellm-model-api-base: http://10.0.1.50:8000/v1 x-litellm-attempted-retries: 0
19:29:27 req 7 HTTP 200 x-litellm-model-api-base: http://10.0.1.50:8000/v1 x-litellm-attempted-retries: 0
19:29:28 req 8 HTTP 200 x-litellm-model-api-base: http://10.0.1.50:8000/v1 x-litellm-attempted-retries: 0
19:29:28 req 9 HTTP 200 x-litellm-model-api-base: http://10.0.1.50:8000/v1 x-litellm-attempted-retries: 1
19:29:29 req 10 HTTP 200 x-litellm-model-api-base: http://10.0.1.50:8000/v1 x-litellm-attempted-retries: 0
19:29:31 req 11 HTTP 200 x-litellm-model-api-base: http://10.0.1.50:8000/v1 x-litellm-attempted-retries: 0
19:29:32 req 12 HTTP 200 x-litellm-model-api-base: http://10.0.1.50:11434 x-litellm-attempted-retries: 0
chat requests the 429 mock rejected:
19:29:22
19:29:28

Don’t try to shortcut this with the mock_testing_fallbacks request flag that older tutorials use: the reliability docs say the proxy strips it from incoming requests since v1.85.0, so it only works against the Python Router. Remove the flaky-429 entry and the container (docker rm -f mock429) when you are done.

Cache repeat prompts in Redis

The response cache sits in front of the router: an identical request (same model, messages and parameters) is answered from Redis without touching a backend. It is useful for evaluation suites, CI jobs and agents that resend the same system prompt, and harmful for anything that needs a fresh answer each time. The cache reuses the Redis container from the core stack, so it only needs these lines under litellm_settings.

  cache: true
  cache_params:
    type: redis
    host: os.environ/REDIS_HOST
    port: os.environ/REDIS_PORT
    ttl: 600

After a restart, send the same question twice and time both. Keep the body identical, because any change to the messages or parameters produces a different cache key.

for run in first second; do
  curl -s -o /tmp/c.json -D /tmp/c.txt -w "$run request: %{time_total}s\n" http://localhost:4000/v1/chat/completions \
    -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" \
    -d '{"model": "qwen-ollama", "messages": [{"role": "user", "content": "What port does PostgreSQL listen on?"}], "max_tokens": 30}'
  grep -iE '^x-litellm-(cache-key|response-cost):' /tmp/c.txt
done

Redis answered the second request in 54 ms instead of 1.27 seconds. Only the cached response carries x-litellm-cache-key, so that is the header to check when you suspect a stale answer. The x-litellm-response-cost header repeats the original price on the hit, but spend tracking logs the hit at $0, so budgets are not charged for cached answers.

first request: 1.268319s
x-litellm-response-cost: 0.00072
second request: 0.054028s
x-litellm-cache-key: 0876979590aaf79cfe0d0b5579c943367a3c7c985fc3b40e73db00a799783263
x-litellm-response-cost: 0.00072

A client that must bypass the cache for one call adds "cache": {"no-cache": true} to the request body; we used exactly that in the troubleshooting test below.

The cache can be switched on from the UI instead, with no cache lines in the config at all. We did that on the second gateway, whose config had none:

  1. Open Response Cache in the sidebar and switch to the Cache Settings tab.
  2. Leave Redis Type on Node (Single Instance) and type redis in Host, the Compose service name.
  3. Type 6379 in Port.
  4. Click Test Connection.
  5. Click Save Changes.
LiteLLM Response Cache settings tab with the Redis host and port of the Compose stack
  1. 1Cache Settings
  2. 2Host
  3. 3Port
  4. 4Test Connection
  5. 5Save Changes

Running the two-request loop again, with a new question and a grep for the cache key only, confirmed it. The repeat answered in 21 ms instead of 406 ms and carried a cache key, so the UI setting is live without a restart.

for run in first second; do
  curl -s -o /tmp/c.json -D /tmp/c.txt -w "$run request: %{time_total}s\n" http://localhost:4000/v1/chat/completions \
    -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" \
    -d '{"model": "qwen-ollama", "messages": [{"role": "user", "content": "Which port does Redis listen on?"}], "max_tokens": 30}'
  grep -iE "^x-litellm-cache-key:" /tmp/c.txt
done

The first request went to Ollama and the second came straight out of Redis, so only the second one prints a cache key line.

first request: 0.406344s
second request: 0.020555s
x-litellm-cache-key: 08ebd5b42aa1fa37729725373371d1ff55db114f7f76d6b26ea3b1ed49478dfa

Only the second request printed a cache key, which is the same signal as in the config-file test. A client cannot tell which path turned the cache on, because the response headers look the same either way.

Mask PII with the Presidio guardrail

Presidio is Microsoft’s open source PII detector, and LiteLLM calls it as a pre_call guardrail, so emails and phone numbers are replaced before the prompt leaves the gateway. It runs as two extra containers, which makes this section optional: skip it and nothing else in the guide changes. Add the services to compose.yaml.

  presidio-analyzer:
    image: mcr.microsoft.com/presidio-analyzer:latest
    restart: unless-stopped

  presidio-anonymizer:
    image: mcr.microsoft.com/presidio-anonymizer:latest
    restart: unless-stopped

Both images listen on port 3000 inside the Compose network. Tell the proxy where they are by adding two lines to the litellm service’s environment block.

      PRESIDIO_ANALYZER_API_BASE: http://presidio-analyzer:3000
      PRESIDIO_ANONYMIZER_API_BASE: http://presidio-anonymizer:3000

Then define the guardrail in config.yaml as a top-level block. Each entity gets an action. MASK replaces it with a placeholder and lets the request continue, while BLOCK rejects the whole request, so reserve it for data that should never leave your network at all.

guardrails:
  - guardrail_name: presidio-pii
    litellm_params:
      guardrail: presidio
      mode: pre_call
      presidio_language: en
      pii_entities_config:
        EMAIL_ADDRESS: MASK
        PHONE_NUMBER: MASK
        CREDIT_CARD: BLOCK

Run docker compose up -d; the analyzer image is 1.58 GB and used 751 MB of RAM in our lab, so it is the heaviest piece of the stack after the models. As written, the guardrail runs only when a request names it, which keeps it off traffic that doesn’t need it. Ask the model to repeat a line containing an email and a phone number.

curl -s http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" \
  -d '{"model": "qwen-ollama", "guardrails": ["presidio-pii"], "max_tokens": 60,
       "messages": [{"role": "user", "content": "Repeat this line exactly: email [email protected] or call 212-555-0187 about ticket 4411."}]}' \
  | jq -r '.choices[0].message.content'

Neither the email nor the phone number reached the model, although the output suggests otherwise for the second one.

email <EMAIL_ADDRESS> or call 010-1234-5678 about ticket 4411.

Don’t read that 010 number as a leak. The /guardrails/apply_guardrail endpoint shows the text the model actually received, and the small model filled in the <PHONE_NUMBER> placeholder with an invented number.

curl -s http://localhost:4000/guardrails/apply_guardrail \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" \
  -d '{"guardrail_name": "presidio-pii", "text": "email [email protected] or call 212-555-0187 about ticket 4411."}' | jq

Both entities were masked before the prompt left the gateway. Run this check with samples of your own data before trusting the guardrail with it, since Presidio’s recognisers miss formats they weren’t built for.

{
  "response_text": "email <EMAIL_ADDRESS> or call <PHONE_NUMBER> about ticket 4411."
}

Use that endpoint whenever a masked answer looks wrong, because the model’s output is the worst place to judge what the guardrail did. A card number trips the BLOCK rule and the request stops at the gateway with a 400.

curl -s http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" \
  -d '{"model": "qwen-ollama", "guardrails": ["presidio-pii"], "max_tokens": 30,
       "messages": [{"role": "user", "content": "My card is 4111 1111 1111 1111, is it valid?"}]}' | jq .error

Because the message names both the entity and the guardrail, the client knows exactly why it was refused. It arrives as an ordinary 400, so existing error handling catches it without changes. Log the message too, because it is the only place that separates a guardrail refusal from a malformed request.

{
  "message": "Blocked entity detected: CREDIT_CARD by Guardrail: presidio-pii. This entity is not allowed to be used in this request.",
  "type": "invalid_request_error",
  "param": null,
  "code": "400"
}

The guardrail also appears under Guardrails in the UI. To run it on every request without the client asking, add default_on: true to its litellm_params, which the open source tier supports. Attaching a guardrail to one key or team only, including through the Guardrails field of the virtual key dialog, is the Enterprise “guardrails per key/team” feature, so on the open source tier the choice is all traffic or opt-in per request.

LiteLLM Guardrails page listing the presidio-pii guardrail loaded from config.yaml

That list shows the config-file guardrail. To build the same guardrail in the UI instead, click Add New Guardrail, choose Add Provider Guardrail, and fill the first step.

  1. Type presidio-ui in Guardrail Name.
  2. Select Presidio PII in Guardrail Provider.
  3. Select pre_call in Mode. Mode is a multi-select, so check that pre_call is the only chip before you click Next.
Create guardrail dialog in LiteLLM with the name presidio-ui, the Presidio PII provider and pre_call mode
  1. 1Guardrail Name
  2. 2Guardrail Provider
  3. 3Mode

We left the Presidio API base fields empty, because the proxy already has PRESIDIO_ANALYZER_API_BASE and PRESIDIO_ANONYMIZER_API_BASE in its environment. In our scripted browser run, clicks on the Mode list did not select anything, while the arrow keys and Enter did. The second step lists every PII type with a checkbox and an action.

  1. Tick CREDIT CARD and change its action to BLOCK.
  2. Tick Email Address and PHONE NUMBER, leaving both on MASK.
  3. Click Create Guardrail.
Presidio PII configuration step in LiteLLM with credit card set to BLOCK and email address set to MASK
  1. 1BLOCK
  2. 2Email Address
  3. 3Create Guardrail

The UI guardrail answered both tests exactly like the config one. Apply it to the sample line, then send the card number with "guardrails": ["presidio-ui"] in the request.

curl -s http://localhost:4000/guardrails/apply_guardrail \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" \
  -d '{"guardrail_name": "presidio-ui", "text": "email [email protected] or call 212-555-0187 about ticket 4411."}' | jq

The card test is the chat request from earlier with the new guardrail name. Only that name changes, so the body is otherwise identical to the config-file test.

curl -s http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" \
  -d '{"model": "qwen-ollama", "guardrails": ["presidio-ui"], "max_tokens": 30,
       "messages": [{"role": "user", "content": "My card is 4111 1111 1111 1111, is it valid?"}]}' | jq .error

The mask came back first. The second call was refused with the same 400 that the config-file guardrail returned, so clients see no difference between the two.

{
  "response_text": "email <EMAIL_ADDRESS> or call <PHONE_NUMBER> about ticket 4411."
}
{
  "message": "Blocked entity detected: CREDIT_CARD by Guardrail: presidio-ui. This entity is not allowed to be used in this request.",
  "type": "invalid_request_error",
  "param": null,
  "code": "400"
}

Both results arrived without a proxy restart, so a guardrail added in the UI takes effect as soon as you save it.

Expose MCP tools through the gateway

The MCP gateway lets agents reach tool servers through the same port and the same virtual keys as the models, instead of each agent holding its own list of MCP URLs and tokens. It is in the open source tier. We pointed it at DeepWiki’s public MCP server, which needs no auth, by adding a top-level block to config.yaml.

mcp_servers:
  deepwiki:
    url: https://mcp.deepwiki.com/mcp
    transport: http

After a restart, the REST endpoint lists the tools the gateway discovered. It is LiteLLM’s own wrapper rather than MCP, which makes it handy for scripts and health checks.

curl -s http://localhost:4000/mcp-rest/tools/list \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -r '.tools[] | "\(.name)  \(.description[0:70])"'

DeepWiki exposes three tools, and the gateway found all of them without any credentials.

ask_wiki_question  Ask any question about a GitHub repository's codebase and get an AI-po
read_wiki_contents  View documentation about a GitHub repository.
read_wiki_structure  Get a list of documentation topics for a GitHub repository.

An MCP client talks JSON-RPC to /mcp/ and authenticates with x-litellm-api-key. The same listing over that endpoint returns the tool names prefixed with the server name, so two servers can each have a search tool without colliding.

curl -s http://localhost:4000/mcp/ \
  -H "x-litellm-api-key: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" -H "Accept: application/json, text/event-stream" \
  -d '{"jsonrpc": "2.0", "id": 1, "method": "tools/list", "params": {}}' | sed -n 's/^data: //p' | jq -r '.result.tools[].name'

An agent sees the prefixed names, so write any tool allow lists in your agent config with the prefix included.

deepwiki-ask_wiki_question
deepwiki-read_wiki_contents
deepwiki-read_wiki_structure

A tool call goes through the gateway the same way, here asking DeepWiki for the structure of the LiteLLM repository itself.

curl -s http://localhost:4000/mcp-rest/tools/call \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" \
  -d '{"server_id": "deepwiki", "name": "read_wiki_structure", "arguments": {"repoName": "BerriAI/litellm"}}' \
  | jq -r '.content[0].text' | head -12

DeepWiki returned its real page list for the repository. The call left from the gateway, so the client never needed DeepWiki’s URL.

Available pages for BerriAI/litellm:

- 1 Overview
  - 1.1 Getting Started and Repo Orientation
  - 1.2 System Architecture at a Glance
  - 1.3 Cookbook, Examples, and Local Testing
- 2 Core SDK
  - 2.1 Completion Functions and Core APIs
  - 2.2 Responses API
  - 2.3 Router and Load Balancing
    - 2.3.1 Routing Strategies
  - 2.4 Provider Integrations

Open MCP Servers in the UI and the config-file server shows as Healthy over HTTP. The same page adds servers to the database without editing the config, which suits a server you are only trying out. We added DeepWiki that way on the second gateway, whose config had no mcp_servers block. Click Add New MCP Server, then + Custom Server, and fill the form.

  1. Type deepwiki in MCP Server Name; the alias fills itself in.
  2. Choose Streamable HTTP in Transport Type.
  3. Enter https://mcp.deepwiki.com/mcp in MCP Server URL.
Add New MCP Server form in the LiteLLM Admin UI with the server name deepwiki, Streamable HTTP transport and the DeepWiki URL
  1. 1MCP Server Name
  2. 2Transport Type
  3. 3MCP Server URL

Further down the form, set Authentication to None before you click Add MCP Server. Left empty, the form refuses to save with “Please select an auth type”, even for a server that needs no credentials. The management API then lists the UI server.

curl -s http://localhost:4000/v1/mcp/server -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  | jq -c '.[] | {server_name, url, transport, auth_type}'

A second call to the tools endpoint, printing names only, shows what the gateway discovered behind it. Names alone keep the output short enough to compare with the list from the config-file server above.

curl -s http://localhost:4000/mcp-rest/tools/list -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq -r '.tools[] | .name'

The first line is the stored record, and its auth_type of none matches the form. The three names after it are the tools the gateway discovered.

{"server_name":"deepwiki","url":"https://mcp.deepwiki.com/mcp","transport":"http","auth_type":"none"}
ask_wiki_question
read_wiki_contents
read_wiki_structure

Tool discovery worked straight after the save, so a UI-added server is usable at once. This is the health list for the config-file server from the first gateway:

LiteLLM MCP Servers page showing the DeepWiki server as healthy over HTTP

One default to change early: with no MCP list on a key or team, every key can reach every server. Restrict a key with object_permission ({"mcp_servers": ["deepwiki"]}) on /key/generate, or set require_key_mcp_access_defined: true under general_settings so the team list becomes a ceiling instead of a default. Stdio servers are off unless the proxy runs with LITELLM_ENABLE_MCP_STDIO=true, which we’d leave off on a shared gateway.

Scrape LiteLLM metrics with Prometheus

Prometheus is the piece that tells you a backend is cooling down before users notice. Two details decide whether it works with several workers. Each worker writes its counters to a shared directory, set with PROMETHEUS_MULTIPROC_DIR. A dedicated metrics listener on port 4001 serves them without virtual-key auth, so you never put a key in prometheus.yml. This is the one feature in the guide with no Admin UI path in v1.106.6: the Add Callback list under Settings, Logging & Alerts offers Datadog, Langfuse, OpenTelemetry, S3 and others, but not Prometheus, so it is configured in files only. Make three changes to the litellm service in compose.yaml. First, the command gains the metrics port.

    command: ["--config", "/app/config.yaml", "--port", "4000", "--num_workers", "4", "--prometheus_metrics_port", "4001"]

The environment block gains the multiprocess directory, and a tmpfs mount gives it a clean directory on every start.

      PROMETHEUS_MULTIPROC_DIR: /tmp/prometheus_multiproc
    tmpfs:
      - /tmp/prometheus_multiproc

Add a Prometheus service, and promdata: under the top-level volumes. That volume keeps the scraped history across restarts, and the image is pinned to the release we ran for the same reason as the LiteLLM tag.

  prometheus:
    image: prom/prometheus:v3.15.1 # https://github.com/prometheus/prometheus/releases
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - promdata:/prometheus
    restart: unless-stopped

Port 4001 is deliberately not published, so only containers on the Compose network can scrape it. Create the scrape config.

vim ~/litellm/prometheus.yml

A single job pointing at the metrics port is enough, because the multiprocess directory already merges all four workers into one endpoint.

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: litellm
    static_configs:
      - targets: ["litellm:4001"]

Finally, turn the callback on in config.yaml under litellm_settings. It is the only Prometheus setting that lives in config.yaml, since the rest sits in compose.yaml and prometheus.yml.

  callbacks: ["prometheus"]

Apply with docker compose up -d; the proxy log prints Serving Prometheus metrics on 0.0.0.0:4001/metrics when the listener starts. Check the target and one counter through the Prometheus API.

curl -s http://localhost:9090/api/v1/targets | jq -r '.data.activeTargets[] | "\(.labels.job) \(.scrapeUrl) \(.health)"'
curl -s 'http://localhost:9090/api/v1/query' --data-urlencode 'query=sum by (requested_model) (litellm_proxy_total_requests_metric_total)' \
  | jq -r '.data.result[] | "\(.metric.requested_model)\t\(.value[1])"'

Prometheus reports the target up, and requests are already split by the model name clients asked for, MCP tool calls included.

litellm http://litellm:4001/metrics up
qwen-small	16
qwen-vllm	1
qwen-ollama	5
MCP: read_wiki_structure	1
null	1

Per-deployment metrics matter most for the wiring, since they tell you which backend is failing rather than that something is. After the 429 test, this query separates failures and cooldowns by backend.

for m in litellm_deployment_failure_responses_total litellm_deployment_cooled_down_total litellm_deployment_successful_fallbacks_total; do
  echo "== $m"
  curl -s http://localhost:9090/api/v1/query --data-urlencode "query=sum by (api_base, exception_status) ($m)" \
    | jq -r '.data.result[] | "\(.metric.api_base // "-")  status=\(.metric.exception_status // "-")  \(.value[1])"'
done

The flaky backend shows four 429s across our two runs and two cooldowns. The fallback counter is empty because the stopped-vLLM fallback happened before the last restart, and the tmpfs directory starts clean, so counters reset with the container.

== litellm_deployment_failure_responses_total
http://10.0.1.50:8099/v1/chat/completions  status=429  4
== litellm_deployment_cooled_down_total
http://10.0.1.50:8099/v1  status=429  2
== litellm_deployment_successful_fallbacks_total

Alert on increase(litellm_deployment_cooled_down_total[5m]) > 0 per api_base and on litellm_deployment_state above 0 (1 is a partial outage, 2 a complete one). Budget burn has its own gauges, litellm_remaining_team_budget_metric and litellm_remaining_api_key_budget_metric. To chart them, point Grafana at this Prometheus as in our Prometheus and Grafana guide.

Tune the proxy for production

The gateway is cheap next to the models it fronts. With four workers, the LiteLLM container held 3.04 GiB, roughly 640 MB per worker, plus a 185 MB metrics process, while PostgreSQL used 102 MB and Redis 6 MB. The production docs size it as 1 vCPU and 4 GiB per worker, with 4 GiB as a floor, and say to set workers equal to vCPUs on a VM or run one worker per pod on Kubernetes and scale pods. Size by request rate, not by model size: LiteLLM’s own benchmark reached 1,170 RPS on four 4-vCPU instances against a mock backend, so a team sending tens of requests a second needs one small instance and a second only for availability. Warm, the proxy added little. We sent eight requests to Ollama after the restart and read the overhead header on each.

for i in $(seq 1 8); do
  curl -s -o /dev/null -D - http://localhost:4000/v1/chat/completions \
    -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" \
    -d "{\"model\": \"qwen-ollama\", \"messages\": [{\"role\": \"user\", \"content\": \"Overhead probe $RANDOM\"}], \"max_tokens\": 3}" \
    | grep -iE '^x-litellm-(overhead-duration-ms|response-duration-ms)' | tr -d '\r' | tr '\n' ' '; echo
done

Six of the eight overheads sat between 12 and 56 ms. The two outliers of 459 and 683 ms came from four workers, Presidio and two model servers sharing four vCPUs, which is the strongest argument for keeping the gateway off the GPU box.

x-litellm-response-duration-ms: 470.96896171569824 x-litellm-overhead-duration-ms: 28.564
x-litellm-response-duration-ms: 230.7281494140625 x-litellm-overhead-duration-ms: 23.9432
x-litellm-response-duration-ms: 185.6989860534668 x-litellm-overhead-duration-ms: 12.486
x-litellm-response-duration-ms: 850.4199981689453 x-litellm-overhead-duration-ms: 683.0082
x-litellm-response-duration-ms: 622.7719783782959 x-litellm-overhead-duration-ms: 459.384
x-litellm-response-duration-ms: 190.2909278869629 x-litellm-overhead-duration-ms: 17.4038
x-litellm-response-duration-ms: 238.6770248413086 x-litellm-overhead-duration-ms: 56.0479
x-litellm-response-duration-ms: 268.4938907623291 x-litellm-overhead-duration-ms: 18.8689 

We added two database settings from the production docs under general_settings. proxy_batch_write_at writes spend every 60 seconds instead of on each request, and the connection pool caps each worker at 10 PostgreSQL connections. Divide your PostgreSQL max_connections by instances times workers to pick that number.

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  proxy_batch_write_at: 60
  database_connection_pool_limit: 10

You pay for the batching with the budget overshoot shown earlier, because the longer the batch interval, the further /key/info trails the live counter. For health checks, wire the three endpoints to the right consumers.

curl -s http://localhost:4000/health/liveliness; echo
curl -s http://localhost:4000/health/readiness | jq '{status, db, cache, litellm_version}'
curl -s http://localhost:4000/health -H "Authorization: Bearer $LITELLM_MASTER_KEY" | jq '{healthy_count, unhealthy_count, unhealthy: [.unhealthy_endpoints[]?.model]}'

Liveliness is for the container restart policy, readiness (which returns 503 when the database is configured but unreachable, unless allow_requests_on_db_unavailable is on) is for the load balancer, and /health probes every model, so keep that one on a slow schedule.

"I'm alive!"
{
  "status": "healthy",
  "db": "connected",
  "cache": null,
  "litellm_version": null
}
{
  "healthy_count": 5,
  "unhealthy_count": 0,
  "unhealthy": []
}

Beyond those, the production page recommends a few switches we’d set before the first real client, and one we’d avoid:

  • LITELLM_LOG=ERROR and json_logs: true under litellm_settings, so logs are parseable and quiet
  • disable_env_credential_login: true once a named admin exists, as the UI banner asks
  • DISABLE_SCHEMA_UPDATE=true on the proxy when you run several instances, with migrations run once as a separate job
  • Avoid allow_requests_on_db_unavailable: true unless the proxy is unreachable from the internet, because the docs say it lets requests through without the database checks

Pin the image tag, as the LITELLM_VERSION line in .env does, and upgrade on purpose: the project ships stable patch releases for several minor lines at once, and its release notes list breaking “action required” changes in most minor versions.

Point Claude Code and SDK clients at the proxy

Any OpenAI SDK reaches the gateway by changing two values, the base URL (http://10.0.1.50:4000/v1) and the API key (a virtual key), and the model becomes the public name from model_list. Aider already routes through the LiteLLM SDK internally, so the Aider cheat sheet model strings map straight onto these names. The proxy also speaks the Anthropic Messages API on /v1/messages, which is what Claude Code uses.

curl -s http://localhost:4000/v1/messages \
  -H "x-api-key: $LITELLM_MASTER_KEY" -H "anthropic-version: 2023-06-01" -H "Content-Type: application/json" \
  -d '{"model": "qwen-vllm", "max_tokens": 30, "messages": [{"role": "user", "content": "Say hello from vLLM in five words."}]}' | jq '{type, model, content, stop_reason, usage}'

vLLM only speaks the OpenAI API, so the Anthropic-shaped answer below is the gateway’s translation. Usage arrives as input_tokens and output_tokens, the field names Anthropic clients expect.

{
  "type": "message",
  "model": "qwen-vllm",
  "content": [
    {
      "type": "text",
      "text": "Hello! How can I assist you today?"
    }
  ],
  "stop_reason": "end_turn",
  "usage": {
    "input_tokens": 39,
    "output_tokens": 10
  }
}

Claude Code needs the base URL, a token and the model names. We used the master key in the lab, as below, but give a real agent its own virtual key that allows the model it calls (ui-demo above allows qwen-vllm), so it is limited to that key’s models and budget. We ran it in print mode against both backends, with the small-model variable that the current docs mark deprecated in favour of ANTHROPIC_DEFAULT_HAIKU_MODEL.

export ANTHROPIC_BASE_URL=http://localhost:4000
export ANTHROPIC_AUTH_TOKEN=$LITELLM_MASTER_KEY
export ANTHROPIC_MODEL=qwen-vllm
export ANTHROPIC_SMALL_FAST_MODEL=qwen-vllm
export MAX_THINKING_TOKENS=0
claude -p "Reply with the single word: pong"

The round trip worked and the answer was useless, and both outcomes have a cause you can fix. Claude Code warns that it doesn’t recognise the model and assumes a 200k-token window, then sends a prompt far bigger than our 4,096-token vLLM model accepts.

Claude Code doesn't recognize the model "qwen-vllm", so it assumes a context window of 200k tokens. Update Claude Code, or see https://code.claude.com/docs/en/model-config#correct-the-window-for-a-gateway-or-custom-model-id to set the window.
[claude-code:unrecognized_model] {"model":"qwen-vllm","query_source":"sdk"}
Great! Let's proceed with the reminder to make some small changes. Can you provide the changes you'd like to apply?

The Logs page explains the nonsense. vLLM rejected the oversized prompt, the fallback sent it to Ollama, and Ollama answered from a truncated prompt of about 2,050 tokens. The first attempt, with qwen-ollama as the model, failed outright with "qwen2.5:0.5b" does not support thinking until we set MAX_THINKING_TOKENS=0.

LiteLLM request logs with per-request cost, duration, model and the Claude Code calls that fell back to Ollama

So the wiring for Claude Code is proven, but a coding agent needs a backend with a context window in the tens of thousands of tokens and tool-calling support, which means a real model on a GPU. The Claude Code cheat sheet covers the rest of its flags.

These are the failures we hit in the lab, each between the gateway and a backend, with the fix that worked. Two of them looked like success from the client side, which is why they took longest to find.

Fallback succeeds but the answer ignores most of the prompt

This one is silent, which makes it the most dangerous. We sent a 6,000-token prompt to qwen-vllm. vLLM refused it because its window is 4,096 tokens, the fallback moved it to Ollama, and Ollama quietly cut it to its default context and answered. The client saw HTTP 200. To reproduce it, build a long prompt and send it once straight to vLLM and once through the gateway:

LONG=$(python3 -c 'print("The quick brown fox jumps over the lazy dog. " * 600)')
jq -n --arg c "$LONG Summarise that in five words." '{model: "qwen-vllm", max_tokens: 20, messages: [{role: "user", content: $c}]}' > /tmp/long.json
curl -s http://10.0.1.50:8000/v1/chat/completions -H "Content-Type: application/json" \
  -d "$(jq '.model = "qwen2.5-0.5b-instruct"' /tmp/long.json)" | jq -r .error.message
curl -s -D /tmp/h.txt http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" \
  -d @/tmp/long.json | jq '{model, usage}'
grep -iE '^x-litellm-(model-api-base|attempted-fallbacks)' /tmp/h.txt

vLLM’s error is explicit, yet through the gateway the same prompt “succeeds” with only 2,050 prompt tokens counted.

This model's maximum context length is 4096 tokens. However, your prompt contains at least 4097 input tokens, which exceeds this limit even without any output tokens. Please reduce the length of the input prompt. (parameter=input_tokens, value=4097)
{
  "model": "ollama_chat/qwen2.5:0.5b",
  "usage": {
    "completion_tokens": 11,
    "prompt_tokens": 2050,
    "total_tokens": 2061
  }
}
x-litellm-model-api-base: http://10.0.1.50:11434
x-litellm-attempted-fallbacks: 1

Give the Ollama deployment a context window at least as large as the model it stands in for, so a fallback never sees less of the prompt than the primary would have. Add the num_ctx line to its litellm_params, which LiteLLM passes through to Ollama, and keep the model_info prices as they were so budgets still count this model.

  - model_name: qwen-ollama
    litellm_params:
      model: ollama_chat/qwen2.5:0.5b
      api_base: http://10.0.1.50:11434
      num_ctx: 8192

Re-run with the cache bypassed, or Redis hands back the truncated answer from the first attempt. The cache key is built from the request, not the backend settings, so changing num_ctx alone does not invalidate the stored answer.

jq '.cache = {"no-cache": true}' /tmp/long.json > /tmp/long-nocache.json
curl -s -D /tmp/h.txt http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" \
  -d @/tmp/long-nocache.json | jq '{model, usage}'

All 6,038 tokens now reach the fallback model. The prompt and the fallback agree again, and the client gets an answer to the question it actually sent.

{
  "model": "ollama_chat/qwen2.5:0.5b",
  "usage": {
    "completion_tokens": 11,
    "prompt_tokens": 6038,
    "total_tokens": 6049
  }
}

For prompts that are simply too long for the primary, LiteLLM also has context_window_fallbacks under router_settings (it needs enable_pre_call_checks: true), which picks a bigger model before calling instead of after a failure. We didn’t test that path here.

vLLM CPU hangs with “No available shared memory broadcast block found in 60 seconds”

With --dtype bfloat16 on our AVX2-only Intel i7-6700, vLLM logged its KV cache size, finished “Warming up”, then printed this line every minute for 19 minutes while one worker sat at 100% CPU and the API never opened. There was no error line. Switching to --dtype float32 brought it up in 329 seconds. The vLLM CPU docs list avx512f as recommended and AVX2 as limited, so check lscpu | grep -o avx512f before choosing a dtype; on AVX-512 hardware bfloat16 is the documented default.

Error: “qwen2.5:0.5b” does not support thinking

Claude Code sends a thinking parameter by default, Ollama rejects it for models without reasoning support, and LiteLLM passes the 400 through. Set MAX_THINKING_TOKENS=0 in the agent’s environment, as we did, or point the agent at a model that supports thinking. Claude Code’s environment variable docs also list CLAUDE_CODE_DISABLE_THINKING=1 for gateways that reject the parameter. The proxy log adds a useful hint when this happens: it names the model groups that have fallbacks configured and says the failing group had none.

Requests to a backend time out after the host renews its address

Halfway through the lab, every call to 10.0.1.50 started timing out after 10 seconds while localhost:8000 still answered. An automatic restart of systemd-networkd had dropped the secondary address we added by hand. If your backends sit on an address you configured manually, make it persistent (a netplan file on Ubuntu) and remember that the gateway’s cooldown will mark the deployment unhealthy within two failures, so the symptom often shows first as traffic piling onto the other backend.

Keep reading

Ollama Models Cheat Sheet: Best Models by VRAM, Tested (PDF) AI Ollama Models Cheat Sheet: Best Models by VRAM, Tested (PDF) Open Source LLM Comparison Table (2026) AI Open Source LLM Comparison Table (2026) Claude Code Cheat Sheet: Commands, Shortcuts and Flags (PDF) AI Claude Code Cheat Sheet: Commands, Shortcuts and Flags (PDF) Dual GPU LLM Build: PCIe x8/x8 vs x16/x4, PSU and Cooling AI Dual GPU LLM Build: PCIe x8/x8 vs x16/x4, PSU and Cooling OpenCloud vs ownCloud vs Nextcloud: Tested on Identical VMs Cloud OpenCloud vs ownCloud vs Nextcloud: Tested on Identical VMs Install Dockge on Ubuntu: Self-Hosted Docker Compose Manager Containers Install Dockge on Ubuntu: Self-Hosted Docker Compose Manager

Leave a Comment

Press ESC to close