🏆 Live Soccer TV: BerriAI/litellm/pull/

mateo-berri · 2026-11-11T19:11:45Z

Handwritten 1-3 sentence description

Google launched the Gemini Decisions API today, and LiteLLM users can now call gemini/gemini-decision-1 on the proxy's /v1/decisions and /v1/systemone routes with spend tracked. Before this PR the proxy answered 400 with "Unknown Decisions provider 'gemini'"

Details

TLDR

Problem this solves:

gemini/ is not a decisions provider, so both routes return 400
Gemini Decision 1 launched today with no LiteLLM support

How it solves it:

New GeminiDecisionsConfig posts to /v1beta/decisions with x-goog-api-key
Maps yes/no, choice and score questions plus images and inline audio/video
Adds the gemini/gemini-decision-1 cost map row at $0.10 per 1M input, $0 output, with Google's published rate limits

User Flow

Before: a developer with a gemini/gemini-decision-1 deployment cannot use it

They send POST http://localhost:4000/v1/decisions with a predicate, a choice and a score question
The proxy answers 400 Unknown Decisions provider 'gemini', same for /v1/systemone

After: the same request returns calibrated answers and records spend

They send the same POST http://localhost:4000/v1/decisions
The proxy answers 200 with a probability per question, 0 output tokens, and x-litellm-response-cost of about $0.0000183

Pre-Submission checklist

I have added meaningful tests
The handful of test files covering my change pass locally
My PR passes all required CI/CD checks
My PR's scope is as isolated as possible; it only solves 1 specific problem
Every automated review on my PR's latest commit has finished, and I have addressed or replied to each of its findings before requesting a maintainer review

Screenshots / Proof of Fix

Client destination overrides cannot inherit deployment keys

Greptile found that the earlier credential guard covered environment keys but could still receive a key already merged from the deployment. The router now asks the provider to validate the original caller connection before merging deployment credentials. The same validation still runs for direct SDK calls

Before at 3114ba07a7, both /v1/decisions and /v1/systemone sent a fake deployment-key canary to a local receiver when the client supplied api_base or base_url without api_key. After at 2d082b5800, all four requests return HTTP 400 requiring an explicit key, and the receiver gets zero requests. Both routes still return HTTP 200 when the caller supplies its own key, and the receiver sees only that key

The live reproduction uses the receiver and commands in the credential section below, with one change to its gemini-default deployment, adding api_key: shared-google-key-canary under litellm_params. Each run used two serving workers and one dedicated Redis shared by those workers. Final worker PIDs were 33031, 33032. A second live run at the same 2d082b5800 tip used the Shared vault Gemini early-access QA credential through the gemini-real model configured below. Both routes returned HTTP 200 with answers and 92 input tokens each. That run again rejected all four deployment-key override requests with zero outbound calls, and both explicit-key requests succeeded. Its two worker PIDs were 33821, 33822. The four new regression cases fail on the previous code and pass with the fix

At 2d082b5800, all 47 focused Gemini and call tests pass, including normal administrator-configured destinations and explicit caller keys. The 377 tests in tests/unit/decisions/test_main.py pass in the freshly built environment. make check passes in full. Commands were PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 .venv/bin/python -m pytest -p pytest_asyncio.plugin -p respx.plugin tests/unit/decisions/test_call.py tests/unit/llms/gemini/decisions/test_transformation.py -q --tb=short, the same pytest invocation for tests/unit/decisions/test_main.py, and UV_PROJECT_ENVIRONMENT=$PWD/.venv make check

Required checks pass at 2d082b5800. Greptile reports 5/5 with no open findings, Bugbot reports no new issues, and all review threads are resolved

CI repair

The original unit passed failure came from the metrics-server occupied-port test expecting the child-exit message, even though startup can correctly fail through its timeout instead. Both occupied-port tests now require MetricsServerStartupError without depending on which failure wins the race. The branch also includes the upstream Rust beta-header fix from #51272 through a merge of main. That upstream fix left two identical arms for Compact20260904, so the duplicate is removed to pass Rust lint

The original failure is recorded in the proxy-infra-root job. Its error was Prometheus metrics server did not answer ... within 30s, while the test required exited with code ...

E2E exception for the CI-only changes at 3114ba07a7. That follow-up removes a timing-dependent message assertion from two existing tests, merges the upstream branch and removes an unreachable duplicate Rust match arm with identical behavior. The production metrics-server startup code is unchanged. PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 PYTHONPATH=$PWD python -m pytest -p pytest_asyncio.plugin tests/unit/proxy/test_prometheus_metrics_server.py -q --tb=short passes all 12 tests, including live local child processes and occupied sockets. ruff check tests/unit/proxy/test_prometheus_metrics_server.py and ruff format --check tests/unit/proxy/test_prometheus_metrics_server.py both pass. cargo clippy --manifest-path litellm-rust/Cargo.toml -p litellm-llms-types --all-targets --locked -- -D warnings passes. cargo test --manifest-path litellm-rust/Cargo.toml -p litellm-llms-types --locked providers::anthropic::beta::tests passes all 111 tests, including the original failing config-matrix test. The provider-flow proof below covers the Gemini changes

Custom Gemini destinations require their own key

At 3e10a869f5, requests that set api_base or base_url must also supply api_key. This prevents the shared GEMINI_API_KEY or GOOGLE_API_KEY from being sent to the requested host. The default Google destination still uses environment credentials, and an administrator-configured GEMINI_API_BASE still works with its environment key

The proxy already rejects client-side connection overrides by default. This reproduction enables allow_client_side_credentials to exercise the affected path. It uses a fake environment-key canary and a local HTTP receiver, plus separate real Gemini calls using the Shared vault's early-access QA credential

Before at 7be92ec774, both routes sent the environment-key canary to the custom host for api_base and base_url. After at 3e10a869f5, all four requests return HTTP 400 with supply an explicit api_key, and the receiver gets zero requests. On both revisions, an explicit custom key reaches only the custom receiver, and both normal Gemini routes return HTTP 200 with answers and usage

Each proxy ran with two serving workers and its own Redis. After worker PIDs were 2218, 2219. Commands used --num_workers 2 --detailed_debug --reload --use_v2_migration_resolver

To reproduce, start a receiver on port 4011 that records the x-goog-api-key header and returns {"answers":{"ok":{"type":"noul","noul":0.9}}}. Export GEMINI_API_KEY=shared-google-key-canary, set QA_GEMINI_KEY to a real Gemini key, and start Redis on port 6398. Use this proxy config

model_list:
  - model_name: gemini-default
    litellm_params:
      model: gemini/gemini-decision-1
  - model_name: gemini-real
    litellm_params:
      model: gemini/gemini-decision-1
      api_key: os.environ/QA_GEMINI_KEY
general_settings:
  master_key: sk-local-auth-proof
  allow_client_side_credentials: true
  coordination_redis:
    host: 127.0.0.1
    port: 6398
router_settings:
  redis_host: 127.0.0.1
  redis_port: 6398

Run the receiver in a separate terminal

from http.server import BaseHTTPRequestHandler, HTTPServer

class Receiver(BaseHTTPRequestHandler):
    def do_POST(self):
        self.rfile.read(int(self.headers.get("Content-Length", "0")))
        print("canary received", self.headers.get("x-goog-api-key") == "shared-google-key-canary", flush=True)
        payload = b'{"answers":{"ok":{"type":"noul","noul":0.9}}}'
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(payload)))
        self.end_headers()
        self.wfile.write(payload)

HTTPServer(("127.0.0.1", 4011), Receiver).serve_forever()

redis-server --bind 127.0.0.1 --port 6398 --save '' --appendonly no

python litellm/proxy/proxy_cli.py --config config.yaml --num_workers 2 --detailed_debug --reload --use_v2_migration_resolver --port 4000
curl http://127.0.0.1:4000/v1/systemone -H 'Authorization: Bearer sk-local-auth-proof' -H 'Content-Type: application/json' -d '{"model":"gemini-default","api_base":"http://127.0.0.1:4011","state":"A mug arrived cracked.","questions":{"ok":{"type":"noul","instructions":"Is the mug damaged?"}}}'
curl http://127.0.0.1:4000/v1/decisions -H 'Authorization: Bearer sk-local-auth-proof' -H 'Content-Type: application/json' -d '{"model":"gemini-default","api_base":"http://127.0.0.1:4011","input":"A mug arrived cracked.","questions":[{"name":"ok","type":"predicate","instructions":"Is the mug damaged?"}]}'

Repeat with base_url replacing api_base. Add "api_key":"custom-gateway-key" to verify the explicit-key case returns 200 and the receiver sees that key. For real Google calls, use "model":"gemini-real" and remove the destination override

Regression verification used uv run --no-sync pytest tests/unit/llms/gemini/decisions/test_transformation.py tests/unit/decisions/test_call.py -q. The eight rejection cases fail on the previous implementation and pass with the guard. All 39 focused tests pass. The same tests cover default environment keys, configured destinations, custom explicit keys and bounded image downloads

The repository gate also found two existing Final annotations inside the image-download loops. The local image variable now uses its inferred string type, permitted by the loop-local assignment rule. The focused image-budget tests pass with these annotations

The full make check passed at 3e10a869f5. The annotation-only follow-up at 13f453cd3d passes all 39 focused tests, Ruff, the type-discipline check and targeted type checking. The live proof above ran at 3e10a869f5

Remote image limits and ordering

Gemini requests accept at most 16 images and 20 MiB of decoded image data. These are proxy safety limits. Downloads run one at a time within the remaining byte budget, including inline images already present. Cached images are checked against the same budget and reused. Captions stay beside their images in message inputs

The fix also repairs logging for remote images, which previously tried to parse the original URL as base64 and returned HTTP 500

Live before/after proof ran against e1de5bd and the exact latest tip 7be92ec, with two serving workers and one dedicated Redis per proxy. The focused suite passes all 29 tests. The larger combined suite has an existing test-order failure for the Gemini cost-map entry on both revisions

Both routes changed from HTTP 500 to HTTP 200 for valid remote images. Seventeen images now return HTTP 400 before any image download. Three 11 MiB images return HTTP 400 during the second download, before fetching the third. A decisions request with alternating captions and two images returns HTTP 200 with 2271 input tokens

Recorded results

[
  {
    "phase": "final-before",
    "results": [
      {
        "route": "systemone",
        "case": "small",
        "status": 500,
        "image_fetches": 1
      },
      {
        "route": "systemone",
        "case": "count",
        "status": 500,
        "image_fetches": 31
      },
      {
        "route": "systemone",
        "case": "large",
        "status": 500,
        "image_fetches": 9
      },
      {
        "route": "decisions",
        "case": "small",
        "status": 500,
        "image_fetches": 1
      },
      {
        "route": "decisions",
        "case": "count",
        "status": 500,
        "image_fetches": 31
      },
      {
        "route": "decisions",
        "case": "large",
        "status": 500,
        "image_fetches": 9
      },
      {
        "route": "decisions",
        "case": "paired",
        "status": 500,
        "image_fetches": 2
      }
    ]
  },
  {
    "phase": "verified",
    "results": [
      {
        "route": "systemone",
        "case": "small",
        "status": 200,
        "image_fetches": 1
      },
      {
        "route": "systemone",
        "case": "count",
        "status": 400,
        "image_fetches": 0
      },
      {
        "route": "systemone",
        "case": "large",
        "status": 400,
        "image_fetches": 2
      },
      {
        "route": "decisions",
        "case": "small",
        "status": 200,
        "image_fetches": 1
      },
      {
        "route": "decisions",
        "case": "count",
        "status": 400,
        "image_fetches": 0
      },
      {
        "route": "decisions",
        "case": "large",
        "status": 400,
        "image_fetches": 2
      },
      {
        "route": "decisions",
        "case": "paired",
        "status": 200,
        "image_fetches": 2
      }
    ]
  }
]

Live reproduction commands and client

Install each checkout's dependencies and native extension, plus requests and PyYAML. Export a real GEMINI_API_KEY and save the client below as proof.py. The client starts an image server, Redis and the proxy, sends the requests, then stops those processes. The local image server requires the test config's URL-validation override

python proof.py /path/to/before-checkout final-before
python proof.py /path/to/after-checkout verified

The proxy command used --num_workers 2 --detailed_debug --reload --use_v2_migration_resolver. Both logs confirmed two serving worker PIDs. Credential loading below uses the environment in place of the private QA-vault lookup used locally

import base64, json, os, socket, subprocess, sys, threading, time, urllib.request
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
import requests, yaml
ROOT=Path(sys.argv[1]); LABEL=sys.argv[2]
PYTHON=sys.executable
def port():
    with socket.socket() as s:
        s.bind(('127.0.0.1',0)); return s.getsockname()[1]
proxy_port, image_port, redis_port = port(),port(),port()
key=os.environ["GEMINI_API_KEY"]
env={**os.environ,'GEMINI_API_KEY':key,'LITELLM_LOCAL_MODEL_COST_MAP':'True','PYTHONPATH':str(ROOT)}
png=base64.b64decode('iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAwMCAO+jwL8AAAAASUVORK5CYII=')
seen=[]
class ImageServer(ThreadingHTTPServer):
    request_queue_size=128
class Images(BaseHTTPRequestHandler):
    def log_message(self,*args): pass
    def do_GET(self):
        seen.append(self.path)
        length=11*1024*1024 if '/large/' in self.path else len(png)
        self.send_response(200);self.send_header('Content-Type','image/png');self.send_header('Content-Length',str(length));self.end_headers()
        try:
            self.wfile.write(png)
            for offset in range(len(png),length,8192): self.wfile.write(b'\0'*min(8192,length-offset))
        except (BrokenPipeError,ConnectionResetError): pass
server=ImageServer(('127.0.0.1',image_port),Images)
threading.Thread(target=server.serve_forever,daemon=True).start()
config={'model_list':[{'model_name':'gemini/gemini-decision-1','litellm_params':{'model':'gemini/gemini-decision-1','api_key':'os.environ/GEMINI_API_KEY'}}], 'litellm_settings':{'user_url_validation':False},'general_settings':{'master_key':'sk-image-limit-qa','coordination_redis':{'host':'127.0.0.1','port':redis_port}},'router_settings':{'redis_host':'127.0.0.1','redis_port':redis_port}}
config_path=Path('/tmp/51220-'+LABEL+'.yaml');config_path.write_text(yaml.safe_dump(config))
log_path=Path('/tmp/51220-'+LABEL+'-proxy.log')
logfile=open(log_path,'w');os.chmod(log_path,0o600)
redis=subprocess.Popen(['/opt/homebrew/bin/redis-server','--bind','127.0.0.1','--port',str(redis_port),'--save','','--appendonly','no'],stdout=subprocess.DEVNULL,stderr=subprocess.DEVNULL)
command=[PYTHON,'litellm/proxy/proxy_cli.py','--config',str(config_path),'--num_workers','2','--detailed_debug','--reload','--use_v2_migration_resolver','--port',str(proxy_port)]
proxy=subprocess.Popen(command,cwd=ROOT,env=env,stdout=logfile,stderr=subprocess.STDOUT,start_new_session=True)
results=[]
try:
    for _ in range(180):
        try:
            if requests.get(f'http://127.0.0.1:{proxy_port}/health/liveliness',timeout=1).status_code==200:break
        except requests.RequestException: pass
        if proxy.poll() is not None: raise RuntimeError('proxy exited; inspect '+str(log_path))
        time.sleep(1)
    else: raise RuntimeError('proxy did not become ready; inspect '+str(log_path))
    for route in ('systemone','decisions'):
        for kind in ('small','count','large','paired'):
            if route == 'systemone' and kind == 'paired': continue
            count={'small':1,'count':17,'large':3,'paired':2}[kind]
            urls=[f'http://127.0.0.1:{image_port}/{kind}/{route}/{n}.png' for n in range(count)]
            body={'model':'gemini/gemini-decision-1'}
            if route=='systemone':body.update(state='Is the image light?',questions={'ok':{'type':'noul','instructions':'Is the image light?'}},images=urls)
            else:body.update(input=[{'role':'user','content':[{'type':'input_text','text':'Is the image light?'}, *[{'type':'input_image','image_url':u} for u in urls]]}],questions=[{'name':'ok','type':'predicate','instructions':'Is the image light?'}])
            if kind == 'paired': body['input']=[{'role':'user','content':[{'type':'input_text','text':'Before'}, {'type':'input_image','image_url':urls[0]}, {'type':'input_text','text':'After'}, {'type':'input_image','image_url':urls[1]}]}]
            n=len(seen)
            response=requests.post(f'http://127.0.0.1:{proxy_port}/v1/{route}',headers={'Authorization':'Bearer sk-image-limit-qa'},json=body,timeout=120)
            result={'route':route,'case':kind,'status':response.status_code,'image_fetches':len(seen)-n,'body':response.text[:900]}
            results.append(result);print(json.dumps(result),flush=True)
    Path('/tmp/51220-'+LABEL+'-results.json').write_text(json.dumps({'label':LABEL,'command':command,'config':config,'results':results},indent=2))
finally:
    import signal
    os.killpg(proxy.pid,signal.SIGTERM) if proxy.poll() is None else None
    redis.terminate();server.shutdown();logfile.close()

Two-worker proxies sharing one Redis on port 6397, no database, Gemini key in GEMINI_API_KEY. Upstream API docs are at https://aistudio.google.com/docs/decisions-api

config.yaml and commands

model_list:
  - model_name: gemini/gemini-decision-1
    litellm_params:
      model: gemini/gemini-decision-1
      api_key: os.environ/GEMINI_API_KEY
litellm_settings:
  drop_params: false
general_settings:
  master_key: sk-gemini-decisions-qa
  coordination_redis:
    host: 127.0.0.1
    port: 6397
router_settings:
  redis_host: 127.0.0.1
  redis_port: 6397

redis-server --bind 127.0.0.1 --port 6397 --save '' --appendonly no
python litellm/proxy/proxy_cli.py --config config.yaml --num_workers 2 --detailed_debug --reload --use_v2_migration_resolver --port 4000

Requests sent to both routes

POST /v1/systemone
{"model": "gemini/gemini-decision-1", "state": "The mug arrived cracked.", "questions": {"damaged": {"type": "noul", "instructions": "Is the mug damaged?"}, "action": {"type": "choice", "instructions": "Choose next action", "criteria": {"replace": "Broken item", "keep": "Undamaged item"}}, "severity": {"type": "score", "instructions": "Rate damage", "criteria": ["None", "Severe"]}}}

POST /v1/decisions
{"model": "gemini/gemini-decision-1", "input": "The mug arrived cracked.", "questions": [{"type": "predicate", "name": "damaged", "instructions": "Is the mug damaged?"}, {"type": "choice", "name": "action", "instructions": "Choose next action", "choices": [{"value": "replace", "description": "Broken item"}, {"value": "keep", "description": "Undamaged item"}]}, {"type": "score", "name": "severity", "instructions": "Rate damage", "levels": [{"label": "None"}, {"label": "Severe"}]}]}

Before (`13089b1`)

POST /v1/systemone returns HTTP 400 litellm.BadRequestError: Unknown Decisions provider 'gemini'
POST /v1/decisions returns the same 400

After (`6896d80`)

Response bodies below show the initial text-only proof at 6896d80, trimmed and rounded for reading. The remote-image proof above covers the latest tip 7be92ec

POST /v1/systemone returns HTTP 200 cost=1.83e-05

{"answers":{"damaged":{"type":"noul","noul":0.9895},"action":{"type":"choice","choice":"replace","probabilities":{"replace":0.9987,"keep":0.0013}},"severity":{"type":"score","score":0.4332}},"usage":{"input_tokens":183,"output_tokens":0}}

POST /v1/decisions returns HTTP 200 cost=1.83e-05

{"model":"gemini/gemini-decision-1","answers":[{"type":"predicate","name":"damaged","probability":0.9895},{"type":"choice","name":"action","choice":"replace","confidence":0.9987},{"type":"score","name":"severity","score":0.4332,"confidence":0.5668}],"usage":{"input_tokens":183,"output_tokens":0,"total_tokens":183}}

The proxy log records response_cost: 1.8299999999999998e-05. SDK systemone, decisions, asystemone and adecisions calls also succeed at 1.81e-05 each

Type

🆕 New Feature

Caveats (if any)

Medium

Remote images have live proxy coverage. Inline audio/video remain unit tested
PDFs and remote audio/video URLs have no shared decisions input type yet
Box questions answer 400, Gemini has no box question
Cost map row overlaps feat(gemini): add gemini-decision-1, the Decisions API model priced at $0.10 per 1M input tokens #51215, whichever merges second rebases

Final Attestation

The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Note

Medium Risk
Touches Decisions auth routing, credential validation, and image fetch budgeting on request paths; incorrect behavior could leak keys or mishandle multimodal payloads.

Overview
Adds GeminiDecisionsConfig so gemini/gemini-decision-1 works on /v1/decisions and /v1/systemone, calling Google’s /v1beta/decisions with x-goog-api-key, mapping noul/choice/score questions plus text/image/audio/video input, and registering pricing/metadata for the new model.

Decisions routing now runs validate_client_connection before credentials merge (router and direct calls): a custom api_base/base_url without an explicit api_key returns 400 so shared Google or deployment keys are not sent to arbitrary hosts. For Gemini, remote images are inlined under 16 images and 20 MiB total decode budget via sequential max_bytes downloads; unsupported-inline logging uses ir_to_systemone_request instead of the upstream wire body. Also drops a duplicate Rust Compact20260904 beta match arm and relaxes flaky Prometheus metrics-server startup assertions.

^{Reviewed by Cursor Bugbot for commit 2d082b5. Bugbot is set up for automated code reviews on this repo. Configure here.}

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

devin-ai-integration · 2026-11-11T19:11:49Z

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

Disable automatic comment, CI, and merge conflict monitoring

veria-ai · 2026-11-11T19:12:05Z

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

Reviews

2d082b5 — No new issues
3114ba0 — No new issues
6b83e79 — No new issues

codspeed · 2026-11-11T19:22:21Z

Merging this PR will not alter performance

1 warning

⚠️ 2 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

✅ 43 untouched benchmarks

_{Comparing litellm_gemini_decisions (2d082b5) with main (0608a41)¹}

No successful run was found on main (a5d04ec) during the generation of this report, so 0608a41 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

codecov · 2026-11-11T19:33:50Z

Codecov Report

❌ Patch coverage is 94.90446% with 8 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/llms/gemini/decisions/transformation.py	90.00%	8 Missing ⚠️

📢 Thoughts on this report? Let us know!

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

mateo-berri · 2026-11-11T21:11:06Z

@greptileai

mateo-berri · 2026-11-11T21:11:08Z

bugbot run

greptile-apps · 2026-11-11T21:14:29Z

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge, with the previous QA gap addressed and no new blocking issue found.

Summary

Adds Gemini Decisions support through /v1/decisions and /v1/systemone, including media conversion, usage and pricing.

Gemini Decisions requests now work through both decision routes.
Gemini custom destinations need a caller-supplied key.
Gemini image requests stop at provider count and size limits.
Startup tests and Rust provider mappings keep the same runtime behavior.

_{Reviews (7) · Last reviewed commit: "fix(gemini): validate caller credentials..." · Reviewed by Greptile}

mateo-berri · 2026-11-11T21:18:49Z

@greptileai

mateo-berri · 2026-11-11T21:18:50Z

bugbot run

cursor