A pod sits in CrashLoopBackOff at 2 a.m. Can an AI read the events, the logs and the spec faster than you can? That is the pitch behind AI Kubernetes troubleshooting, and the two tools in the k8sgpt vs HolmesGPT question answer it very differently. K8sGPT runs fixed analyzers over the cluster and asks a model to explain each finding. HolmesGPT hands the model kubectl and a bash shell and lets it investigate. We wanted to know which one finds the root cause, how long it takes, and what it costs you in tokens.
So we broke a three-node k3s cluster six different ways and pointed both tools at every fault, once with a local model on Ollama and once with Gemini on Vertex AI. That makes 24 runs, each timed, with tokens from our own meter or from the cost line HolmesGPT prints, so this doubles as a K8sGPT tutorial and a HolmesGPT Kubernetes walkthrough you can repeat. I ran all of it on GCP in November 2026 with K8sGPT 0.4.41, HolmesGPT 0.46.0 and k3s v1.37.2+k3s1, and every number below comes from those runs.
K8sGPT vs HolmesGPT: results on six broken workloads
HolmesGPT found the real root cause in all six faults with Gemini and in all six with the local model, although the local model got a few side details wrong. K8sGPT nailed the four faults its analyzers are built for, gave only a generic answer on the misspelled environment variable, and reported “No problems detected” on the NetworkPolicy block with both models. The grades below come from reading every answer against the fault we planted.
| Fault | K8sGPT + Gemini | K8sGPT + Ollama | HolmesGPT + Gemini | HolmesGPT + Ollama |
|---|---|---|---|---|
| CrashLoopBackOff (env var typo) | Partial: “verify POSTGRES_PASSWORD is set” | Symptom only | Root cause, with kubectl set env fix | Root cause |
| ImagePullBackOff (bad tag) | Root cause, suggests an old tag | Root cause, invents a “correct” tag | Root cause, suggests an old tag | Root cause, invents a “correct” tag |
| Pending PVC (missing StorageClass) | Root cause | Root cause | Root cause | Root cause, one fix cannot work |
| Readiness probe 404 | Root cause | Root cause | Root cause, quotes nginx logs | Root cause, one wrong detail |
| NetworkPolicy block | Missed: “No problems detected” | Missed: “No problems detected” | Root cause, with fixed policy YAML | Root cause, blames kube-proxy |
| OOMKilled | Root cause | Root cause | Root cause, compares 256 MiB to the 128Mi limit | Root cause, blames the kubelet for the kill |
Speed and token use tell the other half of the story. K8sGPT sends one short prompt per finding, so a namespace costs a few hundred input tokens. HolmesGPT runs an agent loop that calls five to twelve tools and re-sends the growing transcript on every turn, which pushes a single question to 50,000 to 130,000 prompt tokens. Times are wall clock for the whole command; tokens come from our meter or HolmesGPT’s own cost line.
| Fault | K8sGPT + Gemini | HolmesGPT + Gemini | K8sGPT + Ollama | HolmesGPT + Ollama |
|---|---|---|---|---|
| orders | 15.9 s, 197 in / 168 out, $0.0008 | 39.6 s, 56,429 / 1,725, $0.0231 | 162.9 s, 318 / 1,986 | 218.2 s, 50,438 / 979 |
| catalog | 36.7 s, 464 / 205, $0.0011 | 42.9 s, 86,733 / 1,400, $0.0343 | 255.5 s, 599 / 3,117 | 304.1 s, 96,058 / 1,055 |
| reports | 35.1 s, 315 / 180, $0.0009 | 44.0 s, 97,290 / 1,338, $0.0377 | 305.4 s, 498 / 3,678 | 354.9 s, 103,242 / 1,549 |
| web | 42.5 s, 429 / 239, $0.0012 | 52.0 s, 105,005 / 2,504, $0.0408 | 205.4 s, 663 / 2,510 | 588.1 s, 128,296 / 1,473 |
| payments | 0.2 s, no LLM call | 33.3 s, 80,838 / 1,914, $0.0315 | 0.1 s, no LLM call | 362.7 s, 70,537 / 1,361 |
| batch | 19.1 s, 200 / 117, $0.0006 | 32.5 s, 75,432 / 1,467, $0.0621 | 215.0 s, 320 / 2,566 | 238.0 s, 51,548 / 987 |
| Total | 149.5 s, about $0.005 | 244.3 s, $0.2295 | 1,144.3 s, $0 | 2,066.0 s, $0 |
Read the two tables together and three numbers decide which setup suits you:
- Gemini answers: 15 to 52 seconds per question
- A HolmesGPT investigation on Gemini: about 2 to 6 cents logged, budget 5 to 9
- gpt-oss:20b on 16 CPU cores: about 3 to 10 minutes
Gemini made both tools fast enough to use during an incident. The local model is slow mostly because it spends thousands of reasoning tokens before it answers, so you wait even when the prompt is short. That is fine for a homelab postmortem, but if you are the one holding the pager at 2 a.m., it is too slow to help you.
The test lab
Both tools ran on the k3s control plane node, against the same broken namespaces. A small metering proxy counted the tokens for K8sGPT on both models and for HolmesGPT on Ollama, while HolmesGPT on Gemini went straight to Vertex AI and reported its own tokens. HolmesGPT ran as a separate unprivileged user, for a reason we explain in the section on what broke. The diagram shows how the pieces connect:

We used four Compute Engine VMs on Ubuntu 24.04 in one zone. You can rebuild the same thing on Proxmox or spare mini PCs, because nothing in the comparison depends on GCP apart from the Vertex AI endpoint:
- homelab-k3s, 4 vCPU and 16 GB, control plane at 10.0.1.50
- homelab-k3s-w1 and homelab-k3s-w2, 2 vCPU and 8 GB each
- homelab-ollama, 16 vCPU and 64 GB, no GPU, at 10.0.1.60
- Gemini 3.8 Flash on Vertex AI, global endpoint
The Ollama box is oversized on purpose: gpt-oss:20b needs about 13 GB of RAM and every extra core shortens the long reasoning phase. A GPU with 16 GB of VRAM would cut those minutes to seconds, so treat our Ollama times as the CPU-only worst case.
Step 1: Build a three-node k3s cluster
Any conformant cluster works for this test, including one built with kubeadm on Ubuntu. We used k3s because it gives a working cluster in about a minute and ships with a local-path StorageClass, which matters for the PVC fault later. On the control plane node, pin the release and install the server with a world-readable kubeconfig:
export K3S_VERSION="v1.37.2+k3s1" # pin: version tested in this comparison #https://github.com/k3s-io/k3s/releases
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=${K3S_VERSION} sh -s - server --write-kubeconfig-mode 644
The bundled kubectl reads /etc/rancher/k3s/k3s.yaml on its own, but K8sGPT and other upstream tools only look at ~/.kube/config or $KUBECONFIG. Copy the file for your admin user now so K8sGPT finds the cluster later:
mkdir -p ~/.kube
cp /etc/rancher/k3s/k3s.yaml ~/.kube/config
The workers need the join token, which k3s writes on the server. Print it:
sudo cat /var/lib/rancher/k3s/server/node-token
On each worker, run the agent install with the server address and that token in place of the placeholder:
export K3S_VERSION="v1.37.2+k3s1" # pin: version tested in this comparison #https://github.com/k3s-io/k3s/releases
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=${K3S_VERSION} K3S_URL=https://10.0.1.50:6443 K3S_TOKEN=paste-node-token-here sh -
Back on the control plane, all three nodes should report Ready within a minute. If a worker is missing, check that you can reach port 6443 on the server from it, because that is the port the agent joins on. Off GCP, also open 10250/tcp between nodes for kubectl logs and 8472/udp for flannel VXLAN, or both tools will see failures you did not plant:
kubectl get nodes -o wide
Our cluster came up on containerd from the k3s bundle, with the GCP kernel on every node. Yours will show your own kernel and addresses, but the k3s version should match if you kept the pin:

If you want the longer version with firewall ports and the uninstall script, our guides to k3s on Ubuntu 24.04 and k3s on Ubuntu 26.04 cover both.
Step 2: Run Ollama on a separate box
Keep the local model off the cluster nodes. A 20B model running on CPU will happily eat every core, and you do not want it competing with the workloads you are debugging. On the Ollama VM, install the same release we tested:
export OLLAMA_VERSION="0.40.16" # pin: version tested in this comparison #https://github.com/ollama/ollama/releases
curl -fsSL https://ollama.com/install.sh | OLLAMA_VERSION=${OLLAMA_VERSION} sh
By default Ollama listens only on 127.0.0.1 and uses a short context window. Both tools need the API reachable from the cluster, and HolmesGPT needs a large context because it feeds whole kubectl outputs back to the model. Create a systemd drop-in:
sudo mkdir -p /etc/systemd/system/ollama.service.d
sudo vim /etc/systemd/system/ollama.service.d/override.conf
Add these three settings. The keep-alive stops the model from unloading between runs, which would add a reload to every question:
[Service]
Environment=OLLAMA_HOST=0.0.0.0:11434
Environment=OLLAMA_KEEP_ALIVE=4h
Environment=OLLAMA_CONTEXT_LENGTH=32768
Reload systemd, restart the service and pull the model. We picked gpt-oss:20b because it handles tool calls better than most models that fit in this much memory; our Ollama models cheat sheet lists alternatives by memory size.
sudo systemctl daemon-reload
sudo systemctl restart ollama
ollama pull gpt-oss:20b
Port 11434 has no authentication, so open it only to the cluster subnet in your firewall, never to the internet. On GCP the internal-only rule of the default network already allows it between the VMs, and nothing outside the VPC can reach it.
Step 3: Break the cluster six ways
Each fault lives in its own namespace, so you can point both tools at one problem at a time and your results never bleed into each other. The faults are the six that fill most on-call channels. We kept the manifests in ~/faults on the control plane, readable only by the admin user, which turned out to matter.
mkdir -p ~/faults
vim ~/faults/01-orders-crashloop.yaml
The first fault is a Postgres deployment with one transposed letter in the password variable. The container starts, finds no POSTGRES_PASSWORD and exits, so the pod loops forever:
apiVersion: v1
kind: Namespace
metadata:
name: orders
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: orders-db
namespace: orders
spec:
replicas: 1
selector:
matchLabels:
app: orders-db
template:
metadata:
labels:
app: orders-db
spec:
containers:
- name: postgres
image: postgres:17-alpine
env:
- name: POSTGRES_PASSWROD
value: "s3cret-orders"
- name: POSTGRES_DB
value: orders
The second fault points at an nginx tag that does not exist, which gives ImagePullBackOff:
vim ~/faults/02-catalog-imagepull.yaml
Tag 1.299 looks plausible at a glance, which is exactly how this one slips through review. You would catch it in seconds with kubectl describe pod, so any tool that misses it is not worth your time:
apiVersion: v1
kind: Namespace
metadata:
name: catalog
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: catalog-web
namespace: catalog
spec:
replicas: 2
selector:
matchLabels:
app: catalog-web
template:
metadata:
labels:
app: catalog-web
spec:
containers:
- name: web
image: nginx:1.299-alpine
ports:
- containerPort: 80
The third fault requests a StorageClass called fast-ssd that the cluster does not have, so the claim and its pod stay Pending:
vim ~/faults/03-reports-pvc.yaml
The worker itself is harmless, it only appends a timestamp to the volume every 30 seconds. The default local-path class would have bound this claim, so the name is the whole fault:
apiVersion: v1
kind: Namespace
metadata:
name: reports
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: reports-data
namespace: reports
spec:
accessModes:
- ReadWriteOnce
storageClassName: fast-ssd
resources:
requests:
storage: 5Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: reports-worker
namespace: reports
spec:
replicas: 1
selector:
matchLabels:
app: reports-worker
template:
metadata:
labels:
app: reports-worker
spec:
containers:
- name: worker
image: busybox:1.37
command: ["sh", "-c", "while true; do date >> /data/run.log; sleep 30; done"]
volumeMounts:
- name: data
mountPath: /data
volumes:
- name: data
persistentVolumeClaim:
claimName: reports-data
The fourth fault probes /healthz on a stock nginx image that only serves the welcome page. The pods run, never turn Ready, and the Service ends up with no endpoints:
vim ~/faults/04-web-readiness.yaml
Nothing crashes here, which makes it a good test of whether a tool looks past pod status. You only see the problem as 0/1 in the READY column and a probe event:
apiVersion: v1
kind: Namespace
metadata:
name: web
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: storefront
namespace: web
spec:
replicas: 2
selector:
matchLabels:
app: storefront
template:
metadata:
labels:
app: storefront
spec:
containers:
- name: nginx
image: nginx:1.29-alpine
ports:
- containerPort: 80
readinessProbe:
httpGet:
path: /healthz
port: 80
periodSeconds: 5
failureThreshold: 3
---
apiVersion: v1
kind: Service
metadata:
name: storefront
namespace: web
spec:
selector:
app: storefront
ports:
- port: 80
targetPort: 80
The fifth fault is the hardest one on purpose. Every pod in the payments namespace is Running and healthy, but a NetworkPolicy only admits traffic labelled app=gateway, and the checkout client is labelled app=checkout:
vim ~/faults/05-payments-netpol.yaml
k3s ships an embedded network policy controller, so this block is enforced on a default install and you do not need Calico or Cilium to reproduce it. If your cluster runs a CNI without policy support, the policy is silently ignored and the fault never appears:
apiVersion: v1
kind: Namespace
metadata:
name: payments
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: payments-api
namespace: payments
spec:
replicas: 1
selector:
matchLabels:
app: payments-api
template:
metadata:
labels:
app: payments-api
spec:
containers:
- name: api
image: nginx:1.29-alpine
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: payments-api
namespace: payments
spec:
selector:
app: payments-api
ports:
- port: 80
targetPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: checkout
namespace: payments
spec:
replicas: 1
selector:
matchLabels:
app: checkout
template:
metadata:
labels:
app: checkout
spec:
containers:
- name: checkout
image: curlimages/curl:8.17.0
command: ["sh", "-c", "while true; do curl -sS -m 5 http://payments-api/ -o /dev/null && echo \"$(date) payments-api OK\" || echo \"$(date) payments-api FAILED\"; sleep 10; done"]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: payments-api-ingress
namespace: payments
spec:
podSelector:
matchLabels:
app: payments-api
policyTypes:
- Ingress
ingress:
- from:
- podSelector:
matchLabels:
app: gateway
ports:
- port: 80
The last fault allocates 256 MiB in a container limited to 128Mi, so the kernel OOM killer ends it on every start:
vim ~/faults/06-batch-oom.yaml
Python makes the allocation explicit, so there is no doubt about what the right answer is. A good tool should tell you both numbers, the 256 MiB the code wants and the 128Mi it gets:
apiVersion: v1
kind: Namespace
metadata:
name: batch
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: thumbnailer
namespace: batch
spec:
replicas: 1
selector:
matchLabels:
app: thumbnailer
template:
metadata:
labels:
app: thumbnailer
spec:
containers:
- name: thumbnailer
image: python:3.13-alpine
command: ["python", "-c", "import time; buf = bytearray(256 * 1024 * 1024); print('loaded', len(buf)); time.sleep(3600)"]
resources:
requests:
memory: 64Mi
limits:
memory: 128Mi
Apply the whole directory, give it two minutes, then look at the damage and at the checkout logs:
kubectl apply -f ~/faults/
kubectl get pods -A --field-selector metadata.namespace!=kube-system
kubectl -n payments logs deploy/checkout --tail=4
Five namespaces show an obvious status. The payments pods look perfect and only the checkout log shows the failure, which is what makes that fault interesting:

If you practise for Kubernetes exams, these are the same faults the troubleshooting tasks in our CKA practice questions are built around, so the cluster doubles as a drill environment.
Step 4: Install K8sGPT and add both backends
K8sGPT ships a Debian package on its GitHub releases page together with a checksums file. Download both, verify the package and install it on the control plane:
export K8SGPT_VERSION="0.4.41" # pin: version tested in this comparison #https://github.com/k8sgpt-ai/k8sgpt/releases
cd /tmp
curl -sLO https://github.com/k8sgpt-ai/k8sgpt/releases/download/v${K8SGPT_VERSION}/k8sgpt_amd64.deb
curl -sLO https://github.com/k8sgpt-ai/k8sgpt/releases/download/v${K8SGPT_VERSION}/checksums.txt
grep k8sgpt_amd64.deb checksums.txt | sha256sum -c
sudo apt-get install -y ./k8sgpt_amd64.deb
k8sgpt version
The checksum line should end in OK before apt runs. If you see FAILED instead, delete both files and download again rather than installing, and the version check should then report the build:
k8sgpt_amd64.deb: OK
k8sgpt: 0.4.41 (3644ecf), built at: unknown
Even without a model, K8sGPT is useful. A plain k8sgpt analyze runs the analyzers and lists what is wrong, and on our broken cluster it returned 23 findings in under two seconds:
time k8sgpt analyze
Nine of those 23 are noise, unused or empty ConfigMaps in kube-system, default and the other built-in namespaces. The other 14 are our planted faults: the Deployments, the PVC with the missing StorageClass, the storefront Service with no ready endpoints, and the pods underneath them. The NetworkPolicy fault is nowhere in the list:

Now add the models. For Ollama, K8sGPT has a native backend that takes the server address and model name. We checked this direct route on the catalog namespace and it answered in 3 minutes 43 seconds:
k8sgpt auth add --backend ollama --model gpt-oss:20b --baseurl http://10.0.1.60:11434
For Gemini we used the OpenAI-compatible endpoint that Vertex AI exposes, because it works with the plain openai backend and a short-lived gcloud token. Set your project ID first:
export PROJECT_ID="your-gcp-project-id"
k8sgpt auth add --backend openai --model google/gemini-3.8-flash --baseurl https://aiplatform.googleapis.com/v1/projects/${PROJECT_ID}/locations/global/endpoints/openapi --password "$(gcloud auth print-access-token)"
That access token expires after an hour, so re-run the auth add (after k8sgpt auth remove --backends openai) when calls start failing with 401. K8sGPT also has a dedicated googlevertexai backend if you prefer service account auth. Then ask for an explanation of one namespace:
k8sgpt analyze --explain --no-cache -b openai -n orders
Gemini answered in 15.9 seconds. Notice that it names POSTGRES_PASSWORD as something to verify, but never spots that the variable in the spec is misspelled:

That is the K8sGPT design in one picture. The model only sees the one-line error from the analyzer, “the last termination reason is Error”, and never the container logs or the env list, so it can only guess at common causes. Where that error already contains the answer, as with the missing StorageClass or the OOMKilled reason, the explanation is spot on.
Step 5: Install HolmesGPT as a locked-down user
HolmesGPT is a Python CLI. Our first runs taught us to give it its own Unix user with no sudo and its own kubeconfig, because its bash toolset will read anything the user can read, including your fault manifests and notes. Create the user and copy the k3s kubeconfig over:
sudo apt-get install -y pipx
sudo useradd -m -s /bin/bash sre
sudo mkdir -p /home/sre/.kube
sudo cp /etc/rancher/k3s/k3s.yaml /home/sre/.kube/config
sudo chown -R sre:sre /home/sre/.kube
sudo chmod 600 /home/sre/.kube/config
chmod 700 ~
The k3s kubeconfig is cluster-admin, which is fine for a lab. Because we installed k3s with --write-kubeconfig-mode 644, the original at /etc/rancher/k3s/k3s.yaml also stayed world-readable, so sre could read it too. Leave that flag off anywhere but a throwaway lab. On a real cluster, give the sre user a kubeconfig bound to a read-only ClusterRole instead, since HolmesGPT is meant to investigate, not change things. Next, install the pinned release as that user:
sudo -iu sre
pipx install holmesgpt==0.46.0 #https://github.com/HolmesGPT/holmesgpt/releases
pipx ensurepath
~/.local/bin/holmes version
The version command prints just the release number. You will run every HolmesGPT command below as this sre user, never as your admin account:
0.46.0
Before you ask it anything on k3s, raise the tool memory limit. HolmesGPT wraps every tool command in ulimit -v with a default of 800 MB on x86, and the k3s kubectl binary dies under that limit. Export a larger value in the shell profile of the sre user so every session gets it:
echo 'export TOOL_MEMORY_LIMIT_MB=2048' >> ~/.bashrc
source ~/.bashrc
HolmesGPT talks to models through LiteLLM, so the provider is part of the model name. For Gemini on Vertex AI it reads the project and location from the environment and uses Application Default Credentials. On a GCE VM those come from the attached service account, which needs the roles/aiplatform.user role, and the VM needs the cloud-platform scope, set at creation or while it is stopped, because the default scopes do not cover Vertex AI:
export VERTEXAI_PROJECT="your-gcp-project-id"
export VERTEXAI_LOCATION=global
holmes ask "The orders-db deployment in the orders namespace keeps restarting. What is the root cause?" --model vertex_ai/gemini-3.8-flash --no-interactive --log-costs
Off GCP, run gcloud auth application-default login first. For the local model, point LiteLLM at the Ollama box and use the ollama_chat prefix, which supports tool calls:
export OLLAMA_API_BASE=http://10.0.1.60:11434
holmes ask "The orders-db deployment in the orders namespace keeps restarting. What is the root cause?" --model ollama_chat/gpt-oss:20b --no-interactive --log-costs
The HolmesGPT documentation is upfront that Ollama support is experimental and that tool calling may give inconsistent results. Our runs agree with that warning, as the next step shows.
Step 6: Ask both tools about each fault
We asked both tools the same thing per namespace: K8sGPT got analyze --explain -n <namespace>, and HolmesGPT got one plain question in the words an on-call engineer would use. Here is HolmesGPT with Gemini on the CrashLoopBackOff, the fault where K8sGPT only guessed:

It pulled the container log, saw Postgres complain about the missing superuser password, then read the Deployment and compared the two. The fix it proposed, kubectl set env deployment/orders-db -n orders POSTGRES_PASSWORD=s3cret-orders POSTGRES_PASSWROD-, is exactly what you would type yourself. Total cost was $0.023 for 58,154 tokens over seven tool calls.
On Ollama, gpt-oss:20b reached the same answer with fewer tool calls and a slightly more cautious fix, which was to edit the Deployment by hand. It took 218 seconds instead of 40:

For the other four obvious faults, both tools landed on the right cause with both models, so the difference came down to depth. HolmesGPT with Gemini on the readiness fault quoted the nginx access log showing 404s on /healthz, while K8sGPT only relayed the probe failure from the events. On the OOMKilled fault, HolmesGPT with Gemini compared the 256 MiB allocation in the command against the 128Mi limit and suggested a number, where K8sGPT simply said to raise the limit.
Why does K8sGPT miss the NetworkPolicy block?
The payments fault is where the two designs part ways. The NetworkPolicy and Log analyzers are not in the default active set of K8sGPT, so a normal run sees Running pods and a Service with ready endpoints, and reports nothing. Enabling the NetworkPolicy analyzer alone still finds nothing, because that analyzer only flags a policy with an empty pod selector or one that matches no pods, and ours matches payments-api. Adding the Log analyzer surfaces the curl error, and the explanation then sends you to check Service ports:
k8sgpt analyze --explain --no-cache -b openai -n payments
k8sgpt analyze --explain --no-cache -b openai -n payments --filter NetworkPolicy,Log
Neither run mentions the policy. The default run prints “No problems detected”, and the filtered run shows only the symptom, in 9.1 seconds:

HolmesGPT with Gemini listed the pods, Services and NetworkPolicies in the namespace, read the policy YAML, compared the from selector of the policy against the checkout pod labels and only then confirmed the failure in the checkout log. It wrote out a corrected policy that admits app: checkout. That is the full chain of reasoning a human would follow, done in 33 seconds for three cents:

The local model found the same policy and the same label mismatch, but its explanation said kube-proxy drops the packets. On k3s the embedded network policy controller enforces the rule with iptables, not kube-proxy, and its first suggested fix added a gateway: "true" label that would not match app=gateway anyway. The second option it offered was the correct one.
Step 7: Count the tokens and the cost
Neither CLI reports Ollama token counts in a form you can compare, and K8sGPT reports none at all. We routed every K8sGPT call and every HolmesGPT call to Ollama through a 69-line Python proxy on the control plane that forwards to Ollama or Vertex AI and logs the usage block of each response to a JSONL file. Both tools just get a different base URL, for example http://127.0.0.1:8089/ollama. You do not need the proxy to use either tool, since it only exists so you can compare the numbers. If you want to measure your own runs, the Ollama API returns the same counts in its prompt_eval_count and eval_count fields.
For Gemini, HolmesGPT prints its own cost line through LiteLLM, and its six investigations came to $0.2295 for 501,727 prompt and 10,348 completion tokens. Pricing those tokens at the current Vertex AI rate for Gemini 3.8 Flash ($0.75 per million input and $3.75 per million output until 31 December 2026) gives about $0.415, so the LiteLLM figure is lower. The batch run is the one that matches the full rate exactly, which points to cached input billed at a discount on the other five, but we did not confirm it against a billing export. Budget on the higher number, which puts one investigation at roughly 5 to 9 cents. That rate is a promotion: from 1 January 2027 Vertex AI lists Gemini 3.8 Flash at $1.50 and $7.50 per million, which doubles every Gemini cost in this article.
K8sGPT’s five Gemini explanations used 1,605 input and 909 output tokens in total, about half a cent. Against the $0.2295 that HolmesGPT logged, that is a gap of about 50x, and about 90x at the full rate. That gap is the real trade between the two tools. You can run K8sGPT in a cron job on every cluster without noticing the bill. HolmesGPT is something you call when a human is already looking at a problem. On Ollama, both cost you nothing but time.
What broke in the lab
Two problems cost us a full rerun of every HolmesGPT test, and you will hit the first one on any k3s cluster you point it at. The model output also needed a careful read in a few places.
Error: “fatal error: failed to reserve page summary memory”
Every kubectl call HolmesGPT made on k3s failed with this Go runtime error. HolmesGPT prefixes tool commands with ulimit -v set from TOOL_MEMORY_LIMIT_MB, which defaults to 800 MB on x86. On k3s, /usr/local/bin/kubectl is a symlink to the multicall k3s binary, and Go reserves more virtual address space at start than that limit allows:

The failed run was not free either: the model kept retrying and burned 62,028 prompt tokens and $0.031 to produce nothing. With TOOL_MEMORY_LIMIT_MB=2048 the same question cost $0.010. We only hit this with the k3s binary, but the variable costs nothing to set on any cluster.
HolmesGPT read the answer key off disk
While kubectl was failing, HolmesGPT with Gemini did something clever and unwelcome: it used its bash toolset to list the home directory, found ~/faults/01-orders-crashloop.yaml and our run logs, and read the answer from there. Those runs were useless for a comparison. We moved HolmesGPT to the sre user with a 700 home directory on the admin account and repeated all twelve runs. The lesson applies beyond testing, because whatever the HolmesGPT user can read, the model can read and send to the provider.
Confident details that were wrong
Every root cause from gpt-oss:20b was right, but some supporting details were invented, so you should read the evidence it quotes rather than the summary. Would you have spotted these at 2 a.m.? These are the ones we caught:
- catalog, both tools: a “correct” tag
nginx:1.29.9-alpinethat does not exist - reports: a
kubectl patchof the immutablestorageClassName - web: pods in “the ContainerCreating/Running state”, when they were all Running
- payments: kube-proxy blamed for the drop
- batch: the kubelet named as the OOM killer, not the kernel
Gemini invented nothing in our runs, although it also suggested older but valid nginx tags for the catalog fix, so the old tag is a habit of both models rather than a gpt-oss bug. The bigger cost of the local model is reasoning: gpt-oss:20b spent about 2,000 to 3,700 completion tokens on each K8sGPT namespace that Gemini answered in about 120 to 240, which is why a K8sGPT run on Ollama took about 3 to 5 minutes.
Which one should you run?
Run K8sGPT as the cheap, always-on scanner. Plain k8sgpt analyze without a model is free, finishes in seconds and catches the common failures, and adding --explain with a hosted model costs a fraction of a cent per namespace. Just know that it only explains what its analyzers already flagged, so faults that leave pods Running, like a NetworkPolicy block, stay invisible.
Reach for HolmesGPT when you need the actual root cause. It found all six in our lab, and with a hosted model it did so in under a minute for a few cents. Before you point it at a cluster you care about, set it up the way our second round of runs used it:
- A dedicated Unix user with no sudo
- A kubeconfig bound to a read-only ClusterRole
TOOL_MEMORY_LIMIT_MB=2048on k3s- A hosted model for incidents, Ollama for data that cannot leave
Keep the local model for situations where you can wait several minutes and double-check its details. If you live in the terminal, k9s remains the fastest way to confirm what either tool tells you, and our k0s vs k3s vs MicroK8s comparison helps if you are still choosing the cluster to test them on.