AI

XGBoost on FastAPI vs BentoML with Evidently Data Drift Alerts

FastAPI and BentoML served the same XGBoost credit model within about 6% of each other on a 4 vCPU VM. FastAPI with four uvicorn workers peaked at 198.6 requests per second, and BentoML with four workers peaked at 186.8 (BentoML 1.4.40, measured November 2026 on Ubuntu 26.04). Both hit the CPU wall at 8 concurrent clients, and the p95 gap between them was smaller than the run-to-run spread of either one.

Original content from computingforgeeks.com - post 189238

We found a larger gap in the install line. A plain pip install uvicorn left single-client throughput at 18.3 requests per second instead of 73.0. On the monitoring side, Evidently data drift detection flagged 6 of 20 features in a shifted batch of loan applications. Six columns is 30%, below the 50% dataset-drift default that Evidently ships with, so the Grafana alert in this guide fires on a 0.2 share instead. Every number below comes from one lab run, and every file shown is the file that ran.

The measured result: FastAPI vs BentoML on one 4 vCPU box

Each row is the median of three 30 second hey runs against /predict, sent from a second VM over the internal network. Latencies are in milliseconds, and all 115,895 requests in the fair runs returned HTTP 200. We kept the last two rows, the same FastAPI app on a plain uvicorn install, because that is how most tutorials install it; those two are single runs rather than medians.

ServerClientsreq/sp50p95p99
FastAPI + uvicorn[standard], 4 workers173.013.515.416.6
FastAPI + uvicorn[standard], 4 workers8198.239.369.484.6
FastAPI + uvicorn[standard], 4 workers32198.6152.7278.1331.3
FastAPI + uvicorn[standard], 4 workers64190.4321.1505.4588.9
BentoML 1.4.40, 4 workers167.014.516.718.3
BentoML 1.4.40, 4 workers8182.442.267.774.2
BentoML 1.4.40, 4 workers32185.2162.6266.1283.0
BentoML 1.4.40, 4 workers64186.8328.9503.5553.8
FastAPI + plain uvicorn (h11, asyncio)118.354.956.157.0
FastAPI + plain uvicorn (h11, asyncio)8101.562.1115.9124.9

Read the chart below as two lines that never separate by more than 13 ms at p95. Past 8 clients, extra concurrency only queues, so throughput stays flat between 182 and 199 requests per second while p95 grows roughly in step with the client count. Adding clients beyond the core count measures the queue, not the server. At 64 clients both servers held a median p99 under 590 ms (one FastAPI run reached 803 ms) with zero errors, so neither one fell over under overload; they simply made every request wait longer.

FastAPI and BentoML track each other within about 13 ms at every concurrency; both hit the CPU wall at 8 clients and latency grows linearly after that.FastAPI and BentoML track each other within about 13 ms at every concurrency; both hit the CPU wall at 8 clients and latency grows linearly after that.
p95 latency by concurrency, XGBoost credit model on 4 vCPU

Three readings in these numbers decide how much weight the comparison deserves.

  • Single client: FastAPI answers in 13.5 ms at p50 and BentoML in 14.5 ms. FastAPI also kept a small throughput lead at every concurrency, since its three-run ranges never overlapped those of BentoML (197.1 to 199.0 against 178.7 to 184.8 requests per second at 8 clients).
  • Saturation: the model call itself costs about 12 ms per request in process (6.3 to 6.8 ms building the pandas frame, 4.9 to 5.3 ms in predict_proba, over three single-thread runs). Dividing four cores by the measured peak gives roughly 20 ms per request under load, and that budget caps the box near 200 requests per second, whichever framework sits in front.
  • Tail noise: FastAPI p95 at 8 clients ranged from 50.9 to 94.4 ms across its three runs, and BentoML from 48.7 to 72.0 ms. A 1.7 ms difference between the medians sits inside that margin, so treat the p95 columns as a tie.

With a model this cheap, the framework is not the bottleneck; the pandas conversion is. With a model this small (1,000 training rows, about 5 ms in predict_proba), this benchmark measures serving and request overhead, not model compute, so the near tie at p95 may not hold for a heavier model. We did not test one: in principle the fixed per-request overhead would become a smaller share of each request, but that is untested here. Optimizing the feature frame would buy more than switching servers. Building a NumPy array in fixed column order, or scoring through an XGBoost DMatrix directly, skips the per-column categorical rebuild, although we did not benchmark that variant here.

How the test was run

We ran both servers on the same VM, one at a time with the other stopped, so neither competed for CPU. The load generator ran on a separate, smaller VM, since hey on the measured box would have eaten one of the four cores. The diagram shows the full pipeline, including the drift path that runs beside the serving path. Requests never pass through the exporter, so drift checks add nothing to prediction latency, whereas inline checks in the request handler would.

Architecture diagram of the XGBoost model served by FastAPI and BentoML with Evidently drift exporter, Prometheus and Grafana

Our lab setup is listed below so the numbers can be compared with your own hardware.

  • Server VM (bench01): 4 vCPU, 15 GB RAM, Ubuntu 26.04.1, Python 3.14.4. XGBoost 3.4.1 runs with n_jobs=1 per worker, so four workers use four cores without oversubscribing.
  • Load VM (bench02): 2 vCPU, Ubuntu 24.04.5, hey from the Ubuntu archive, with a 10 second warm-up at 8 clients before each set of runs.
  • Payload: one loan applicant with all 20 German Credit features as JSON, and a 46 byte response per request.
  • Monitoring: Evidently 0.7.23 in its own virtualenv, plus Prometheus v3.16.0 and Grafana 13.3.1 in Docker Compose on bench01.

One VM is enough to follow the guide. A second VM only matters when you want load numbers that exclude the CPU used by the load generator itself.

Step 1: Prepare the VM and download the German Credit data

The packages below cover Python virtualenvs, compilers for any wheel that needs building, Docker with the Compose plugin, and hey for load tests. On Ubuntu 26.04 they all come from the Ubuntu archive (several from universe), so no third-party repository is needed. Install hey on the load VM as well if you use one.

sudo apt-get update
sudo apt-get install -y python3-venv python3-dev build-essential docker.io docker-compose-v2 hey

The model trains on the UCI Statlog German Credit dataset, which holds 1,000 applicants, 20 features and a good or bad credit label under CC BY 4.0. Download the archive and extract it into data/ with the zipfile module, since a minimal Ubuntu image may not have unzip.

mkdir -p ~/credit && cd ~/credit
curl -sLO https://archive.ics.uci.edu/static/public/144/statlog+german+credit+data.zip
python3 -m zipfile -e statlog+german+credit+data.zip data
wc -l data/german.data

Expect a count of 1000. The file is space separated, and the categorical columns use UCI codes such as A11 for a checking account below zero and A46 for an education loan. We use both codes later to build the shifted batch, so keep the UCI page open if you want to decode the others.

Create three virtualenvs. Keeping them apart was not optional in this lab, because Evidently 0.7.23 does not import under pandas 3 while the serving stack runs on pandas 3.1.1. One shared environment would force the servers onto the older pandas for no benefit.

python3 -m venv .venv-api .venv-bento .venv-drift
.venv-api/bin/pip install xgboost==3.4.1 scikit-learn pandas fastapi "uvicorn[standard]==0.54.0"  # https://github.com/dmlc/xgboost/releases
.venv-bento/bin/pip install bentoml==1.4.40 xgboost==3.4.1 scikit-learn pandas  # https://github.com/bentoml/BentoML/releases
.venv-drift/bin/pip install evidently==0.7.23 prometheus-client "pandas<3"  # https://github.com/evidentlyai/evidently/releases

Note the [standard] extra on uvicorn. Without it, uvicorn falls back to the pure Python h11 parser and the asyncio loop, and the last two rows of the results table are what you get.

Step 2: Train the XGBoost credit model

Both servers import one shared module, so the request schema, the column order and the scoring code cannot drift apart between FastAPI and BentoML. If each server carried its own copy, a renamed field in one would score garbage in the other without any error. Open features.py.

vim features.py

Paste the module below. The Applicant Pydantic model doubles as the API schema in both frameworks, and load_model() pins XGBoost to one thread per process.

import json
from pathlib import Path

import pandas as pd
import xgboost as xgb
from pydantic import BaseModel

BASE = Path(__file__).resolve().parent
COLUMNS = [
    "checking_status", "duration_months", "credit_history", "purpose",
    "credit_amount", "savings", "employment_since", "installment_rate",
    "personal_status", "other_debtors", "residence_since", "property",
    "age", "other_installments", "housing", "existing_credits", "job",
    "dependents", "telephone", "foreign_worker",
]
NUMERIC = [
    "duration_months", "credit_amount", "installment_rate",
    "residence_since", "age", "existing_credits", "dependents",
]
CATEGORICAL = [c for c in COLUMNS if c not in NUMERIC]


class Applicant(BaseModel):
    checking_status: str
    duration_months: int
    credit_history: str
    purpose: str
    credit_amount: int
    savings: str
    employment_since: str
    installment_rate: int
    personal_status: str
    other_debtors: str
    residence_since: int
    property: str
    age: int
    other_installments: str
    housing: str
    existing_credits: int
    job: str
    dependents: int
    telephone: str
    foreign_worker: str


def to_frame(rows, categories):
    df = pd.DataFrame(rows, columns=COLUMNS)
    for col in CATEGORICAL:
        df[col] = pd.Categorical(df[col], categories=categories[col])
    return df


def load_model():
    model = xgb.XGBClassifier()
    model.load_model(BASE / "model" / "credit-xgb.json")
    # one thread per worker process; 4 workers x 4 threads oversubscribes a 4 vCPU box
    model.set_params(n_jobs=1)
    categories = json.loads((BASE / "model" / "categories.json").read_text())
    return model, categories


def score(model, categories, applicant: Applicant) -> dict:
    df = to_frame([applicant.model_dump()], categories)
    p_bad = float(model.predict_proba(df)[0, 1])
    return {"probability_bad": round(p_bad, 4), "decision": "review" if p_bad >= 0.5 else "approve"}

We store the categories next to the model because native categorical support in XGBoost maps each category to a code by position. A request whose categories arrive in a different order would score silently wrong, so to_frame() rebuilds them from the saved list every time. That rebuild is also the 6 ms or so of frame cost from the results section.

Now create train.py. It splits 800 rows for training and 200 for a holdout, and saves both as CSV for the drift step.

vim train.py

Add the training script. The split is stratified on the label, so the 30% bad rate holds in both the training rows and the holdout.

import json

import pandas as pd
import xgboost as xgb
from sklearn.metrics import accuracy_score, roc_auc_score
from sklearn.model_selection import train_test_split

from features import BASE, CATEGORICAL, COLUMNS, to_frame

raw = pd.read_csv(BASE / "data" / "german.data", sep=" ", header=None, names=COLUMNS + ["target"])
y = (raw["target"] == 2).astype(int)  # UCI codes 1 = good, 2 = bad
categories = {c: sorted(raw[c].unique().tolist()) for c in CATEGORICAL}
X = to_frame(raw[COLUMNS], categories)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)

model = xgb.XGBClassifier(
    n_estimators=300, max_depth=4, learning_rate=0.05, subsample=0.9,
    colsample_bytree=0.8, enable_categorical=True, tree_method="hist",
    eval_metric="auc", random_state=42,
)
model.fit(X_train, y_train, eval_set=[(X_test, y_test)], verbose=False)

p = model.predict_proba(X_test)[:, 1]
print(f"rows: train={len(X_train)} test={len(X_test)} bad_rate={y.mean():.2f}")
print(f"test AUC={roc_auc_score(y_test, p):.3f} accuracy={accuracy_score(y_test, p >= 0.5):.3f}")

(BASE / "model").mkdir(exist_ok=True)
model.save_model(BASE / "model" / "credit-xgb.json")
(BASE / "model" / "categories.json").write_text(json.dumps(categories, indent=1))

(BASE / "batches").mkdir(exist_ok=True)
ref = X_train.astype({c: str for c in CATEGORICAL}).assign(target=y_train.values)
ref.to_csv(BASE / "batches" / "reference.csv", index=False)
cur = X_test.astype({c: str for c in CATEGORICAL}).assign(target=y_test.values)
cur.to_csv(BASE / "batches" / "holdout.csv", index=False)
print("saved model/credit-xgb.json, batches/reference.csv, batches/holdout.csv")

Run it with the API virtualenv, which already has XGBoost and scikit-learn installed.

.venv-api/bin/python train.py

Training finishes in a few seconds on 800 rows and prints three lines.

rows: train=800 test=200 bad_rate=0.30
test AUC=0.816 accuracy=0.800
saved model/credit-xgb.json, batches/reference.csv, batches/holdout.csv

An AUC of 0.816 is in line with what this dataset usually gives a tuned tree model. The 800 training rows become the Evidently reference set later, which matters because drift is always measured against the data the model learned from. Comparing against last week of traffic instead would hide a slow drift that has been building for months.

Step 3: Serve the model with FastAPI and uvicorn workers

Our FastAPI app is 18 lines long, since the scoring logic lives in features.py. Create app.py.

vim app.py

Define a health route and the prediction route. Body(embed=True) makes the request body {"applicant": {...}}, the same shape BentoML expects, so one payload file tests both servers.

from typing import Annotated

from fastapi import Body, FastAPI

from features import Applicant, load_model, score

model, categories = load_model()
app = FastAPI(title="credit-scorer")


@app.get("/healthz")
def healthz():
    return {"status": "ok"}


@app.post("/predict")
def predict(applicant: Annotated[Applicant, Body(embed=True)]):
    return score(model, categories, applicant)

We declared the route as a plain def rather than async def. FastAPI runs sync routes in a thread pool, which keeps the 12 ms scoring call from blocking the event loop. An async def route calling the same code would serialize every request inside a worker, so four workers would behave like four single-threaded servers.

Save a test applicant as payload.json.

vim payload.json

Every load test in the results table sent this exact payload, a 24 month loan of 4,870 DM.

{"applicant": {"checking_status": "A12", "duration_months": 24, "credit_history": "A32", "purpose": "A43", "credit_amount": 4870, "savings": "A61", "employment_since": "A73", "installment_rate": 3, "personal_status": "A93", "other_debtors": "A101", "residence_since": 4, "property": "A124", "age": 35, "other_installments": "A143", "housing": "A153", "existing_credits": 2, "job": "A173", "dependents": 2, "telephone": "A191", "foreign_worker": "A201"}}

Start uvicorn with four workers, one per vCPU. Each worker is a separate process with its own copy of the model, so memory grows with the worker count while the CPU stays shared fairly between them.

.venv-api/bin/uvicorn app:app --host 0.0.0.0 --port 8000 --workers 4

From a second terminal, send the payload with curl -s -H "Content-Type: application/json" -d @payload.json http://127.0.0.1:8000/predict. The screenshot below shows that call, plus the same call against BentoML on port 3000 after Step 4.

FastAPI on port 8000 and BentoML on port 3000 return the same XGBoost credit prediction

Both servers return a 0.7178 probability of default and a review decision for this applicant. Identical scores confirm that both load the same model file and the same category order, which is the precondition for comparing their latency at all. Only the JSON spacing differs, since BentoML serializes with a space after each colon.

Step 4: Serve the same model with BentoML

Stop uvicorn with Ctrl+C first, so the two servers never share the CPU. A BentoML service is a class in which __init__ loads the model once per worker and each @bentoml.api method becomes an HTTP route. Create service.py.

vim service.py

Point the service at the shared module. The traffic timeout of 10 seconds only matters if a request hangs, since a normal prediction takes about 12 ms in process.

import bentoml

from features import Applicant, load_model, score


@bentoml.service(workers=4, resources={"cpu": "4"}, traffic={"timeout": 10})
class CreditScorer:
    def __init__(self):
        self.model, self.categories = load_model()

    @bentoml.api
    def predict(self, applicant: Applicant) -> dict:
        return score(self.model, self.categories, applicant)

Setting workers=4 is what makes this comparison fair. The BentoML documentation on parallel requests states that the default is one worker and that each extra worker loads its own model copy, the same process model as --workers in uvicorn. Left at the default, BentoML would use one of the four cores and lose the benchmark for a reason that has nothing to do with the framework.

Serve it on port 3000. Each of the four worker processes loads its own model copy at startup, so give it a few seconds before the first request.

.venv-bento/bin/bentoml serve service:CreditScorer --port 3000

BentoML also publishes Prometheus metrics on /metrics at the same port with no extra code, and Step 7 scrapes them. FastAPI needs a separate instrumentation package to do the same, so count that as one more dependency on the FastAPI side.

For deployment, BentoML packages the service into a Bento, a versioned directory with the code, the model files and a locked dependency list that bentoml containerize turns into an image. The build reads bentofile.yaml.

vim bentofile.yaml

List the service, the files to include and the Python packages to pin. Unlisted files stay out of the Bento, which keeps local CSVs and virtualenvs from being packaged.

service: "service:CreditScorer"
labels:
  model: credit-xgb
include:
  - "service.py"
  - "features.py"
  - "model/credit-xgb.json"
  - "model/categories.json"
python:
  packages:
    - xgboost==3.4.1
    - scikit-learn==1.9.1
    - pandas==3.1.1

Run the build from the project directory with .venv-bento/bin/bentoml build, then list the result with .venv-bento/bin/bentoml list.

bentoml build output creating the credit_scorer Bento and bentoml list showing its size

Our Bento came to 581.55 KiB, and its Model Size column reads 0.00 B because the XGBoost JSON travels as an included file instead of going through the BentoML model store. For a model file of 530,766 bytes that is fine. Anything large belongs in the model store, so several Bentos can share one copy rather than each carrying their own.

Step 5: Load test both servers with identical parameters

Fairness comes from running one script against both URLs, so any bias in the method hits both servers equally. Copy payload.json to the load VM, then create bench.sh there.

vim bench.sh

The script warms up for 10 seconds, then runs three 30 second passes at 1, 8, 32 and 64 clients. Each pass sits inside a 60 second timeout, because a stuck hey process would otherwise hang forever; with set -e a timed-out pass stops the script, so a failed series shows up at once instead of leaving a gap in the results.

#!/usr/bin/env bash
# usage: bench.sh <name> <url>   e.g. bench.sh fastapi http://10.128.0.8:8000/predict
set -euo pipefail
NAME=$1 URL=$2
mkdir -p results
hey -z 10s -c 8 -m POST -T application/json -D payload.json "$URL" > /dev/null
for c in 1 8 32 64; do
  for run in 1 2 3; do
    # timeout guards against a hey run that never returns
    timeout 60 hey -z 30s -c "$c" -m POST -T application/json -D payload.json "$URL" > "results/${NAME}-c${c}-r${run}.txt"
    printf '%s c=%-3s run=%s %s\n' "$NAME" "$c" "$run" "$(grep 'Requests/sec' "results/${NAME}-c${c}-r${run}.txt")"
  done
done

Run it once per server, with only that server running on the model VM. Replace 10.0.1.50 with the address of your server (the usage comment in the script still shows our lab address).

chmod +x bench.sh
./bench.sh fastapi http://10.0.1.50:8000/predict
./bench.sh bentoml http://10.0.1.50:3000/predict

Each server takes about 6 minutes. Below is the first of the three FastAPI runs at 8 clients.

hey load test at 8 concurrent clients against the FastAPI XGBoost endpoint with latency distribution

This run did 197.06 requests per second with a p95 of 94.4 ms, while the next two landed at 50.9 and 69.4 ms. Throughput barely moved between runs (197.1 to 199.0), but the tail did, because on a saturated box a single scheduler hiccup stalls 8 queued requests at once. Report medians of several runs, never one.

Four cores divided by about 198 requests per second leaves roughly 20 ms per request, of which about 12 ms is the model path and the rest is HTTP and framework work. Throughput stayed flat from 8 clients up, which points to the cores as the limit, not the network or hey, which is why 32 and 64 clients bought no extra throughput.

Step 6: Detect Evidently data drift on a shifted batch

Drift monitoring compares the features the model sees in production with the features it trained on. To test it, we need one batch that looks like training data and one that does not. The 200 holdout rows serve as the normal batch, and a shifted copy pushes five features at once. The shift mimics a lender that starts marketing larger, longer loans to younger applicants, which is the kind of change a model trained on last year never saw.

  • credit_amount: multiplied by a random factor between 1.5 and 2.2 per row.
  • duration_months: 12 months added to every loan.
  • age: 9 years younger, floored at 19.
  • checking_status and purpose: 60% of rows set to an overdrawn account (A11), and half set to an education loan (A46).

Create the script that writes both batches. It leaves the target column alone on purpose, so the labels stay those of the original applicants.

vim make_batches.py

The shift uses a fixed seed, so your shifted batch matches ours row for row and your p-values should match the table below.

import numpy as np
import pandas as pd

holdout = pd.read_csv("batches/holdout.csv")
rng = np.random.default_rng(7)

shifted = holdout.copy()
shifted["credit_amount"] = (shifted["credit_amount"] * rng.uniform(1.5, 2.2, len(shifted))).round().astype(int)
shifted["duration_months"] = shifted["duration_months"] + 12
shifted["age"] = (shifted["age"] - 9).clip(lower=19)
shifted.loc[rng.random(len(shifted)) < 0.6, "checking_status"] = "A11"
shifted.loc[rng.random(len(shifted)) < 0.5, "purpose"] = "A46"

holdout.to_csv("batches/batch-normal.csv", index=False)
shifted.to_csv("batches/batch-shifted.csv", index=False)
for name, df in (("normal", holdout), ("shifted", shifted)):
    print(f"{name:8} rows={len(df)} credit_amount_median={df.credit_amount.median():.0f} "
          f"duration_median={df.duration_months.median():.0f} age_median={df.age.median():.0f} "
          f"A11_share={(df.checking_status == 'A11').mean():.2f}")

Run it with the drift virtualenv. Any of the three would work here, since the script needs only pandas and NumPy.

.venv-drift/bin/python make_batches.py

The medians show how hard the batch was pushed. Loan amounts nearly doubled and the overdrawn share went from 28% to 69%.

normal   rows=200 credit_amount_median=2372 duration_median=18 age_median=33 A11_share=0.28
shifted  rows=200 credit_amount_median=4356 duration_median=30 age_median=24 A11_share=0.69

Before running Evidently, we checked what this shift does to the decisions of the model. Scoring both batches (score_batches.py in the lab, calling the same to_frame() and predict_proba() as the servers) gave the result below.

normal   mean_p_bad=0.289 review_rate=0.22 actual_bad_rate=0.30
shifted  mean_p_bad=0.561 review_rate=0.61 actual_bad_rate=0.30

Manual reviews jumped from 22% to 61% of applicants, while the true bad rate stayed at 30% because the labels were never touched. In production you would not have those labels for months. Feature drift is the signal you get on day one, which is why the alert keys on it instead of on accuracy.

DataDriftPreset runs one statistical test per column and then counts how many columns drifted. The Evidently data drift preset documentation sets the dataset rule, which calls the whole dataset drifted when at least 50% of columns drift. Per-column tests follow the drift method explainer. When the reference has 1,000 rows or fewer, it picks Kolmogorov Smirnov for numeric columns with more than 5 unique values, chi-square for categorical or low-cardinality ones and a Z-test for binary ones, and flags a column when the p-value falls below 0.05 (the explainer counts exactly 0.05 as drift too, an edge our exporter treats as no drift). On the shifted batch it chose these tests and returned these p-values.

ColumnTestp-valueDrifted
purposechi-square2.39e-188yes
checking_statuschi-square1.16e-36yes
duration_monthsK-S1.24e-30yes
ageK-S2.18e-22yes
credit_amountK-S9.08e-17yes
employment_sincechi-square0.0115yes
other_debtorschi-square0.140no
installment_ratechi-square0.192no
savingschi-square0.197no
personal_statuschi-square0.221no
existing_creditschi-square0.225no
dependentsZ-test0.382no
residence_sincechi-square0.458no
jobchi-square0.496no
foreign_workerZ-test0.558no
credit_historychi-square0.610no
telephoneZ-test0.772no
other_installmentschi-square0.820no
housingchi-square0.980no
propertychi-square0.982no

Four readings come out of this table, and the last one changes how the alert is built.

  • All five shifted features were caught: every one has a p-value below 1e-16, so detection has a wide margin and would survive a much milder shift.
  • Some numeric columns are tested as categories: Evidently treats a numeric column with few unique values as categorical, so installment_rate (values 1 to 4) got chi-square instead of K-S even though it was declared numeric.
  • One false positive: employment_since was not changed, yet it crossed 0.05 at 0.0115. The unshifted holdout batch also flagged two columns (purpose and employment_since), a 10% share. A 0.05 threshold across 20 columns expects about one chance flag per batch, and employment_since was never shifted, so its p-value is the same in both batches: it reflects how the train and holdout split fell, not a single unlucky batch. That is why the alert threshold sits above the normal-batch share.
  • The default verdict says no drift: 6 of 20 columns is 30%, under the 50% rule, so the dataset-level result of the preset is false for a batch that nearly tripled the review rate.

Act on the last point. With 20 features, half of them must move before the default fires, although a real population change rarely touches more than a handful. Alert on the drifted share with a threshold between the noise floor (10% here) and the real shift (30% here); we used 0.2.

Step 7: Export the drift share to Prometheus

Evidently produces reports, not time series, so a small exporter bridges the gap. It reruns the report whenever the current batch file changes and publishes the results as Prometheus gauges on port 9105. In production the batch file would hold the last hour or day of logged requests; in this lab it is whichever CSV we copy into place.

vim drift_exporter.py

Paste the exporter. Reading the report through snapshot.dict() gives one DriftedColumnsCount entry for the share and one ValueDrift entry per column in Evidently 0.7.

import sys
import time
from pathlib import Path

import pandas as pd
from evidently import DataDefinition, Dataset, Report
from evidently.presets import DataDriftPreset
from prometheus_client import Gauge, start_http_server

BASE = Path(__file__).resolve().parent
REFERENCE = BASE / "batches" / "reference.csv"
CURRENT = BASE / "batches" / "current.csv"
NUMERIC = [
    "duration_months", "credit_amount", "installment_rate",
    "residence_since", "age", "existing_credits", "dependents",
]

DRIFT_SHARE = Gauge("credit_drift_share", "Share of feature columns Evidently flags as drifted")
DRIFT_COUNT = Gauge("credit_drift_columns", "Number of feature columns flagged as drifted")
COLUMN_SCORE = Gauge("credit_drift_score", "Per-column drift score from Evidently", ["column", "method"])
COLUMN_DRIFTED = Gauge("credit_drift_detected", "1 when the column crosses its drift threshold", ["column"])
BATCH_ROWS = Gauge("credit_drift_batch_rows", "Rows in the current batch")
LAST_RUN = Gauge("credit_drift_last_run_timestamp_seconds", "Unix time of the last Evidently run")


def load(path, definition):
    df = pd.read_csv(path).drop(columns="target", errors="ignore")
    return df, Dataset.from_pandas(df, data_definition=definition)


def run_report():
    ref_df = pd.read_csv(REFERENCE, nrows=1).drop(columns="target", errors="ignore")
    definition = DataDefinition(
        numerical_columns=NUMERIC,
        categorical_columns=[c for c in ref_df.columns if c not in NUMERIC],
    )
    _, reference = load(REFERENCE, definition)
    current_df, current = load(CURRENT, definition)
    snapshot = Report([DataDriftPreset()]).run(current, reference)

    for metric in snapshot.dict()["metrics"]:
        config = metric["config"]
        if config["type"].endswith("DriftedColumnsCount"):
            DRIFT_COUNT.set(metric["value"]["count"])
            DRIFT_SHARE.set(metric["value"]["share"])
        elif config["type"].endswith("ValueDrift"):
            score, threshold = metric["value"], config["threshold"]
            # p-value tests drift below the threshold, distance metrics drift above it
            drifted = score < threshold if "p_value" in config["method"] else score >= threshold
            COLUMN_SCORE.labels(config["column"], config["method"]).set(score)
            COLUMN_DRIFTED.labels(config["column"]).set(int(drifted))
    BATCH_ROWS.set(len(current_df))
    LAST_RUN.set(time.time())


def main(interval=30):
    start_http_server(9105)
    seen = None
    while True:
        mtime = CURRENT.stat().st_mtime
        if mtime != seen:
            run_report()
            seen = mtime
            print(f"drift run: share={DRIFT_SHARE._value.get():.2f} columns={DRIFT_COUNT._value.get():.0f}", flush=True)
        time.sleep(interval)


if __name__ == "__main__":
    main(int(sys.argv[1]) if len(sys.argv) > 1 else 30)

We drop the target column before the report, since labels arrive late in production and a drift job should not depend on them. Note also that Report.run() takes the current dataset first and the reference second. Swapping them does not raise an error, so a reversed call quietly measures drift in the wrong direction.

Start with the normal batch as the current one, then launch the exporter with a 15 second polling interval.

cp batches/batch-normal.csv batches/current.csv
nohup .venv-drift/bin/python drift_exporter.py 15 > drift.log 2>&1 &

After the first report, swap in the shifted batch, wait for the next poll, and read the gauges that are not zero.

cp batches/batch-shifted.csv batches/current.csv
sleep 20
tail -2 drift.log
curl -s localhost:9105/metrics | grep -E "^credit_drift_(share|columns|detected)" | grep -v " 0.0$"

The screenshot shows the copy, log and metrics commands with their output on the lab VM.

Evidently drift exporter metrics after the shifted batch: share 0.3 and six drifted columns

Two runs appear in the log, 0.10 with 2 columns on the normal batch and 0.30 with 6 on the shifted one. The same six columns from the p-value table carry credit_drift_detected 1.0. Each report on 800 reference rows and 200 current rows took 1.4 to 1.5 seconds in three timed runs, so a 15 second poll is far more frequent than needed, and once per batch window is enough.

Step 8: Graph the drift in Grafana and fire an alert

Prometheus scrapes the exporter and the BentoML metrics, while Grafana reads Prometheus and evaluates the alert. Both run in Docker Compose, and every Grafana object is provisioned from files so the setup can be rebuilt from scratch. If Compose is new to you, the guide to installing Docker Compose on Linux covers the plugin. Create the directory layout first.

mkdir -p ~/credit/monitoring/grafana/provisioning/{datasources,dashboards,alerting} ~/credit/monitoring/grafana/dashboards
cd ~/credit/monitoring

Our Compose file pins both images. Grafana is published on host port 3001 because BentoML already holds 3000, and Prometheus binds to localhost only.

vim compose.yaml

Add both services. The host-gateway entry lets the Prometheus container reach the exporter and BentoML on the host.

services:
  prometheus:
    image: prom/prometheus:v3.16.0
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.retention.time=15d
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-data:/prometheus
    extra_hosts:
      - host.docker.internal:host-gateway
    ports:
      - 127.0.0.1:9090:9090
    restart: unless-stopped

  grafana:
    image: grafana/grafana:13.3.1
    environment:
      GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD:?set GRAFANA_ADMIN_PASSWORD}
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
      - ./grafana/dashboards:/var/lib/grafana/dashboards:ro
      - grafana-data:/var/lib/grafana
    ports:
      - 3001:3000
    restart: unless-stopped

volumes:
  prometheus-data:
  grafana-data:

Port 3001 is published on all interfaces. Our VM had no firewall rule allowing it from outside, and we reached Grafana through an SSH tunnel. Docker publishes ports through its own iptables rules, so a host firewall such as ufw does not block them unless you add rules to the DOCKER-USER chain. If your host has a public address, change the line to 127.0.0.1:3001:3000 or put Grafana behind a TLS reverse proxy, which the Grafana in Docker guide sets up.

Generate the admin password into a .env file readable only by you. Run it in a subshell, because a strict umask left in your session would make every file you create next unreadable to the containers.

(umask 077; echo "GRAFANA_ADMIN_PASSWORD=$(openssl rand -hex 12)" > .env)

Next comes the Prometheus scrape configuration. Compose mounts it read-only into the container.

vim prometheus.yml

Define one job per source, scraped every 15 seconds.

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: credit-drift
    static_configs:
      - targets: ["host.docker.internal:9105"]
  - job_name: credit-bentoml
    static_configs:
      - targets: ["host.docker.internal:3000"]

Give the Grafana datasource a fixed uid, so the dashboard and alert files can reference it by name.

vim grafana/provisioning/datasources/prometheus.yaml

Point it at the Prometheus service name on the Compose network.

apiVersion: 1
datasources:
  - name: Prometheus
    uid: prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true

A dashboard provider tells Grafana where to find dashboard JSON files.

vim grafana/provisioning/dashboards/credit.yaml

Load everything in the mounted dashboards directory into a folder called Credit model.

apiVersion: 1
providers:
  - name: credit
    folder: Credit model
    type: file
    options:
      path: /var/lib/grafana/dashboards

Our dashboard has four panels: the current share, the share over time, a table of drifted columns and the per-column drift strength. Create the JSON file.

vim grafana/dashboards/credit-drift.json

The fourth panel plots -log10 of each p-value, so 1.3 on its axis means p = 0.05 and higher means stronger drift. Raw p-values would not plot at all, as the errors section explains.

{
  "uid": "credit-drift",
  "title": "Credit model drift",
  "refresh": "30s",
  "time": {
    "from": "now-30m",
    "to": "now"
  },
  "schemaVersion": 41,
  "panels": [
    {
      "type": "stat",
      "title": "Drifted column share",
      "id": 1,
      "gridPos": {
        "x": 0,
        "y": 0,
        "w": 6,
        "h": 7
      },
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "targets": [
        {
          "refId": "A",
          "expr": "credit_drift_share"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "percentunit",
          "min": 0,
          "max": 1,
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 0.2
              }
            ]
          }
        },
        "overrides": []
      }
    },
    {
      "type": "timeseries",
      "title": "Drifted column share over time",
      "id": 2,
      "gridPos": {
        "x": 6,
        "y": 0,
        "w": 18,
        "h": 7
      },
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "targets": [
        {
          "refId": "A",
          "expr": "credit_drift_share",
          "legendFormat": "share"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "unit": "percentunit",
          "min": 0,
          "max": 1,
          "custom": {
            "thresholdsStyle": {
              "mode": "line"
            }
          },
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 0.2
              }
            ]
          }
        },
        "overrides": []
      }
    },
    {
      "type": "table",
      "title": "Columns flagged as drifted",
      "id": 3,
      "gridPos": {
        "x": 0,
        "y": 7,
        "w": 10,
        "h": 10
      },
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "targets": [
        {
          "refId": "A",
          "expr": "credit_drift_detected == 1",
          "instant": true,
          "format": "table"
        }
      ],
      "transformations": [
        {
          "id": "organize",
          "options": {
            "excludeByName": {
              "Time": true,
              "__name__": true,
              "instance": true,
              "job": true,
              "Value": true
            }
          }
        }
      ]
    },
    {
      "type": "timeseries",
      "title": "Per-column drift strength, -log10(p-value)",
      "id": 4,
      "gridPos": {
        "x": 10,
        "y": 7,
        "w": 14,
        "h": 10
      },
      "datasource": {
        "type": "prometheus",
        "uid": "prometheus"
      },
      "targets": [
        {
          "refId": "A",
          "expr": "-log10(credit_drift_score{column=~\"credit_amount|duration_months|age|checking_status|purpose|employment_since\"})",
          "legendFormat": "{{column}}"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "custom": {
            "thresholdsStyle": {
              "mode": "line"
            },
            "scaleDistribution": {
              "type": "log",
              "log": 10
            }
          },
          "thresholds": {
            "mode": "absolute",
            "steps": [
              {
                "color": "green",
                "value": null
              },
              {
                "color": "red",
                "value": 1.3
              }
            ]
          }
        },
        "overrides": []
      }
    }
  ]
}

Put the alert rule in its own provisioning file. It queries credit_drift_share, fires when the value is above 0.2, and waits 2 minutes before firing so a single noisy batch does not page anyone.

vim grafana/provisioning/alerting/credit-drift.yaml

Define the rule group. Query A reads the share, and expression C applies the threshold to it.

apiVersion: 1
groups:
  - orgId: 1
    name: credit-drift
    folder: Credit model
    interval: 1m
    rules:
      - uid: credit-drift-share
        title: Credit model feature drift
        condition: C
        data:
          - refId: A
            relativeTimeRange:
              from: 600
              to: 0
            datasourceUid: prometheus
            model:
              refId: A
              expr: credit_drift_share
              instant: true
          - refId: C
            datasourceUid: __expr__
            model:
              refId: C
              type: threshold
              expression: A
              conditions:
                - evaluator:
                    type: gt
                    params: [0.2]
        for: 2m
        noDataState: Alerting
        execErrState: Error
        labels:
          severity: warning
          model: credit-xgb
        annotations:
          summary: "Evidently flags {{ humanizePercentage $values.A.Value }} of credit features as drifted"

We set noDataState: Alerting on purpose. If the exporter dies, the share disappears from Prometheus, and a drift monitor that goes quiet should page just like one that sees drift. If FastAPI was the last server you benchmarked, start .venv-bento/bin/bentoml serve service:CreditScorer --port 3000 again first, otherwise the credit-bentoml target reads down. Then start the stack.

sudo docker compose up -d
sudo docker compose ps --format "table {{.Service}}\t{{.Image}}\t{{.Status}}"

Both containers should be up. A Prometheus container that restarts in a loop points to the permission error covered further down.

SERVICE      IMAGE                     STATUS
grafana      grafana/grafana:13.3.1    Up 21 minutes
prometheus   prom/prometheus:v3.16.0   Up 22 minutes

Confirm Prometheus can reach both targets through the host gateway.

curl -s localhost:9090/api/v1/targets | python3 -c "import json,sys; [print(t['labels']['job'], t['health']) for t in json.load(sys.stdin)['data']['activeTargets']]"

Each job should report up.

credit-bentoml up
credit-drift up

A down target here usually means the service on the host listens on 127.0.0.1 only. Both start_http_server(9105) and bentoml serve bind all interfaces by default, so both answered the container. The second cause is ufw with its default deny on incoming traffic, which drops the scrapes from the Docker bridge to ports 9105 and 3000; our GCP VM had no ufw, so allow the Compose bridge subnet to those two ports if yours does. The same all-interface binding means ports 8000, 3000 and 9105 are open to anyone who can reach the VM and have no authentication, so firewall them on a VM with a public address. Open a tunnel from your workstation with ssh -L 3001:127.0.0.1:3001 [email protected], browse to http://127.0.0.1:3001 and log in as admin with the password from .env. The Credit model folder holds the provisioned dashboard.

Grafana dashboard with Evidently data drift share moving between 10 and 30 percent and the drifted columns table

Our share line reads 30%, drops to 10% and climbs back to 30%, because we swapped the shifted batch in, back out to the normal batch and in again while testing. Each step to 30% crosses the red 0.2 threshold line. The table lists the same six columns as the exporter output. In the bottom right panel, purpose sits far above the others because its p-value was 2.39e-188, while employment_since sits just above the 1.3 line, which marks it as the weakest signal on the board.

Open Alerting, then Alert rules, and select Credit model feature drift to see its state. The numbers on the figure follow the order in which you check the rule.

  1. The state badge reads Firing. The rule went Pending at the first evaluation above 0.2 and fired 2 minutes later, as configured.
  2. The credit_drift_share query is query A, the same metric the dashboard graphs.
  3. The Threshold, Is above 0.2 expression is condition C, the line between the 10% noise floor and the 30% real shift.
  4. The Pending period 2m is the for: 2m from the provisioning file.
Grafana alert rule Credit model feature drift firing on credit_drift_share above 0.2
  1. 1Firing
  2. 2credit_drift_share query
  3. 3Threshold, Is above 0.2
  4. 4Pending period 2m

With the share at 0.3, humanizePercentage turns the summary into “Evidently flags 30% of credit features as drifted” in the notification. In this lab the alert went to the default notification policy, whose built-in email contact point still holds a placeholder address, so the alert was routed but never delivered. Add an email, Slack or webhook contact point under Alerting, then Contact points, before relying on it. For host metrics on the same box, the Prometheus and Grafana server monitoring guide adds node_exporter to this stack, and the Blackbox Exporter uptime guide can probe /healthz from outside.

Errors we hit in the lab

Four failures cost time during this build. Each has a one-line fix once the cause is known, but none of them names its cause in the error output.

uvicorn –workers 4 stuck at 18 requests per second for one client

Our first FastAPI run used pip install uvicorn without extras. One client got 18.26 requests per second with a p50 of 54.9 ms, against 13.5 ms for the same code later, because every keep-alive request waited about 40 ms on top of the 12 ms model call. Installing uvicorn[standard]==0.54.0, which adds httptools 0.9.0 and uvloop 0.23.0, removed the stall and quadrupled single-client throughput to 73.0. At 8 clients the gain was smaller (101.5 to 198.2 requests per second) but still roughly double. Check what uvicorn picked with .venv-api/bin/pip list | grep -E "httptools|uvloop"; if both are missing, so is half your throughput.

ImportError: cannot import name ‘infer_dtype_from_object’ from ‘pandas.core.dtypes.common’

Evidently 0.7.23 fails on import when pip resolves pandas 3, because its legacy test module imports a pandas internal that pandas 3 removed. We hit it on Python 3.14, and the traceback points at pandas rather than the interpreter. Pinning pandas<3 in the drift virtualenv fixed it.

Evidently 0.7.23 ImportError infer_dtype_from_object under pandas 3 and the pandas below 3 fix

pip settled on pandas 2.3.3. Separate virtualenvs exist in this guide for exactly this reason, since the serving stack stays on pandas 3.1.1 while the drift job pins what Evidently can import.

Error loading config: open /etc/prometheus/prometheus.yml: permission denied

Prometheus exited on start with this message because prometheus.yml and the Grafana provisioning files had mode 600. They had been created under a 077 umask, and the Prometheus container runs as the unprivileged nobody user, which cannot read a file only its owner may open. Make the mounted files world readable with chmod -R a+rX prometheus.yml grafana, then run sudo docker compose restart. Only .env should stay at 600, since Compose reads it on the host and no container ever opens it.

Per-column p-value panel empty on a log scale

Our first version of the bottom right panel graphed raw p-values on a logarithmic axis and showed nothing, because values down to 2.39e-188 sit far outside what the axis renders usefully. Graphing -log10(credit_drift_score) instead turned them into readable numbers, up to 187.6 for purpose on the shifted batch. Drift now reads as up, and the 0.05 cutoff becomes a fixed line at 1.3.

FastAPI or BentoML: which one to run

On raw speed for a small tabular model, pick either. FastAPI led by 1 ms at one client and by about 6% on peak throughput, and both margins are smaller than what a missing [standard] extra or an unpinned thread count costs. The decision rests on everything around the request.

  • FastAPI fits when the model sits inside a larger API with auth, database calls and other routes, when the team already ships FastAPI, or when you want the 13.5 ms floor and will add metrics yourself.
  • BentoML fits when the service is the model: bentoml build gave a versioned 581.55 KiB artifact with locked dependencies, /metrics came free, and the same service class scales to adaptive batching or GPU workers without a rewrite.

For a team already running models in Kubernetes, a model server such as the one in the KServe and vLLM guide changes the picture again. The drift exporter above works in front of any of them, because it reads logged features rather than talking to the server. Our MLOps books list covers the wider practice of shipping and watching models.

Whichever server you pick, keep the drift path. In this lab it was the only signal that caught a population change on day one, since the review rate went from 22% to 61% while accuracy would have stayed unknown until the loans matured. Watch credit_drift_share against the noise floor of your own normal batches, and set the alert between the two.

Keep reading

Ollama Models Cheat Sheet: Best Models by VRAM, Tested (PDF) AI Ollama Models Cheat Sheet: Best Models by VRAM, Tested (PDF) Open Source LLM Comparison Table (2026) AI Open Source LLM Comparison Table (2026) Claude Code Cheat Sheet: Commands, Shortcuts and Flags (PDF) AI Claude Code Cheat Sheet: Commands, Shortcuts and Flags (PDF) Dual GPU LLM Build: PCIe x8/x8 vs x16/x4, PSU and Cooling AI Dual GPU LLM Build: PCIe x8/x8 vs x16/x4, PSU and Cooling MCP vs RAG vs Agent Skills: When to Use Each AI MCP vs RAG vs Agent Skills: When to Use Each Setup Maltrail Malicious Traffic Detection on Linux Monitoring Setup Maltrail Malicious Traffic Detection on Linux

Leave a Comment

Press ESC to close