You have a Linux box with plenty of RAM, no GPU, and you want Hermes Agent running against a model that never leaves the machine. Two endpoints matter here, and most of the confusion we hit came from mixing them up. Hermes calls a model endpoint (Ollama on port 11434, or vLLM on 8000), and Hermes also serves its own OpenAI-compatible API on port 8642 for curl and Open WebUI.
Below we wire both sides of the Hermes Agent Ollama setup on a CPU-only host and record the timings, because on CPU the numbers decide whether the agent is usable at all. Every step ran on an 8 vCPU, 31 GB cloud VM with no GPU. We confirmed the chain working on Ubuntu 26.04 with Hermes Agent 0.23.1 and Ollama 0.40.16 in November 2026.
Keep the two endpoints apart
Open WebUI (or curl) talks to the Hermes API server, Hermes runs its agent loop and tools, and every model call goes to Ollama, with vLLM as an optional second backend on the same box. The diagram below is the whole wiring, numbered in the order a request travels.

When something breaks, the first question is always which hop failed: the client to Hermes, or Hermes to the model. Each hop has its own port, its own key and its own log, so we test them one at a time in the sections that follow, model side first, then the API server, then the browser.
Check how much RAM the 64K window needs
Hermes Agent refuses any model whose context window is under 64,000 tokens, and that one rule sets the memory budget. A small model at 64K needs far more RAM than its download size suggests, because the KV cache for the whole window is allocated up front. We read these figures on the lab host from ollama ps and the vLLM startup log.
| What runs | Memory measured | What drives it |
|---|---|---|
| qwen3:4b-instruct Q4_K_M in Ollama, 4,096 context | 3.2 GB | Ollama’s default window, too small for Hermes |
| qwen3:4b-instruct Q4_K_M in Ollama, 65,536 context | 12 GB | KV cache for the full 64K window |
| llama3.2:3b in Ollama, 65,536 context | 10 GB | Same window, smaller model |
| Qwen3-4B-Instruct in vLLM CPU | 10.76 GiB weights plus 12 GiB KV cache | Unquantized weights, VLLM_CPU_KVCACHE_SPACE=12 |
| Hermes gateway (API server) | 232 MB | Python process, no model inside |
Treat our 8 vCPU and 31 GB as the floor for one user, not a production size. Prefill on CPU is compute-bound, so more cores shorten the cold runs, and each extra concurrent session needs its own KV space: vLLM reported room for just 1.33 full 64K requests at once with 12 GiB of cache. If you are picking between models first, our open source LLM comparison covers the families. Our Xeon reported avx512f in lscpu; check the instruction sets your CPU offers before you try the vLLM backend.
Install Ollama and Hermes Agent
Start on the model side. Ollama’s install script sets up a systemd service that listens on 127.0.0.1:11434, which is exactly where Hermes will look for it later. Loopback is the right place for it here, because only Hermes on the same host needs to reach the model, and nothing else should.
curl -fsSL https://ollama.com/install.sh | sh
We pulled the Qwen3 4B instruct build at Q4_K_M as the main model, plus two smaller ones to compare later. The instruct build matters on a CPU host, because it answers tool calls directly instead of spending slow decode time on a thinking block first.
ollama pull qwen3:4b-instruct-2507-q4_K_M
ollama pull llama3.2:3b
ollama pull qwen3:1.7b
Now the agent side. The Hermes installer runs as your normal user and puts everything under ~/.hermes, including config.yaml, the .env file and the logs we read later. On a headless server we skip the browser and computer-use extras, since neither does anything without a desktop and both add install time.
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash -s -- --non-interactive --skip-browser --skip-computer-use
source ~/.bashrc
Confirm both binaries answer before wiring anything, since a missing PATH entry is easier to spot now than after a failed agent run.
hermes --version
ollama --version
Our lab returned the Hermes release line, install details and the Ollama version:
Hermes Agent v0.23.1 (2026.9.24) · stable · local 3b4215785882
Install directory: /home/eugene/.hermes/hermes-agent
Install method: git
Python: 3.14.9
OpenAI SDK: 2.54.0
Up to date
ollama version is 0.40.16
For the day-to-day Ollama commands (pull, ps, show, rm) keep the Ollama commands cheat sheet open, and use the Ollama models cheat sheet when you pick a different model tag. Both sides are installed now, but they do not know about each other yet.
Open WebUI and the optional vLLM backend both run as containers later on, so install Docker now from the Ubuntu archive and add your user to the docker group. The newgrp line applies the group in the current shell without logging out:
sudo apt-get install -y docker.io
sudo usermod -aG docker $USER
newgrp docker
The Ubuntu package was enough for both containers in this guide, so there is no need to add Docker’s own repository for a lab like this one.
Set Ollama to a 64K context window
Everyone skips this step, and Hermes punishes it. On a host with no GPU, Ollama serves every model at 4,096 tokens unless told otherwise, even when the model itself supports 262,144. Hermes’ system prompt and tool schemas alone are bigger than that, so the very first message fails with a “conversation has grown too long” error. In our lab, ollama ps showed the Qwen model at a 4096 CONTEXT and 3.2 GB before the change, and at 65536 and 12 GB after it.
The fix belongs on the Ollama side: set OLLAMA_CONTEXT_LENGTH=65536 in a systemd drop-in for ollama.service, using the drop-in in the Ollama section of our Hermes Agent cheat sheet. We tested only its first Environment= line, the context length. The other three (flash attention, a q8_0 KV cache, one loaded model) are optional and change the memory and timing numbers in this article. We picked 65536 over the bare 64000 minimum so the window is a power of two with a little margin. It applies to every model Ollama loads, so each one reserves RAM for the full window. Run daemon-reload, restart Ollama, then confirm the variable reached the service, because a model that was already loaded keeps its old window until the restart unloads it:
systemctl show ollama -p Environment
With only the context line added, the variable shows up in the environment line like this. If it is missing, the drop-in file name or the [Service] header is wrong, and Ollama keeps serving 4K:
Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin OLLAMA_CONTEXT_LENGTH=65536
The cheat sheet covers the rest of the Ollama tuning for Hermes. On a CPU host, 64K is the only setting you can’t skip, and the next section proves it with the first real reply.
Point Hermes at the Ollama endpoint
Now we connect the agent to the model. The source of truth is ~/.hermes/config.yaml, and the custom provider with an explicit base_url is the wiring that worked for us. Picking the ollama provider without a URL, or exporting OPENAI_BASE_URL while the provider is custom, both left Hermes with nothing to call (the errors are in the troubleshooting section). The interactive hermes model command is the other route; per the providers docs it writes the provider, base_url and model to config.yaml (plus an API key for endpoints that need one), but we did not run it in this lab. The hermes config set lines below are the path we tested.
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:11434/v1
hermes config set model.default qwen3:4b-instruct-2507-q4_K_M
hermes config set model.api_key ollama
hermes config set model.context_length 65536
Ollama ignores the API key, and we set a placeholder so the endpoint carries an explicit credential rather than relying on Hermes’ keyless default. The context_length line tells Hermes what window the server really gives it, and it should match the drop-in. Read the block back:
hermes config get model
All five keys should be present, with the API key masked in the output:
default: qwen3:4b-instruct-2507-q4_K_M
provider: custom
base_url: http://127.0.0.1:11434/v1
api_key: '***'
context_length: 65536
Send a one-shot prompt with time in front, then check what Ollama loaded. Be patient on the first one, because ours took almost nine minutes with no output at all while the CPU worked through the prompt, and the usage numbers after the output explain where that time went.
time hermes -z "Reply with the word ready"
ollama ps
The reply came back, and ollama ps confirms the 64K window and the 12 GB footprint, all of it on CPU:
ready
real 8m51.965s
user 0m0.914s
sys 0m0.093s
NAME ID SIZE PROCESSOR CONTEXT RUNNER UNTIL
qwen3:4b-instruct-2507-q4_K_M 0edcdef34593 12 GB 100% CPU 65536 llamacpp 4 minutes from now
Nearly nine minutes for one word looks broken, but it is not. With every default toolset enabled, Hermes sent about 11,600 tokens of system prompt and tool schemas with that two-word prompt, and the CPU had to prefill all of them. The same prompt run straight after took 2.9 seconds, because Ollama reused 11,575 of those tokens from its cache.

Ollama’s prompt cache is the reason the agent stays usable on CPU. Anything that changes the start of the prompt (a new toolset, a different system prompt, another model loaded in between) throws it away, and you pay the full prefill again on the next request, so keep one model loaded and one toolset per platform.
Disable extra toolsets and measure the models
On CPU, the cheapest speedup is fewer tools. Every enabled toolset adds schemas to each request, and a server agent rarely needs a browser, image generation or vision. We limited the CLI runs to the terminal and file toolsets with -t, and used --yolo so the terminal command runs without the approval prompt in one-shot mode.
time hermes -m qwen3:4b-instruct-2507-q4_K_M -t terminal,file --yolo --usage-file usage-qwen4i-cold.json -z "Run uname -r with the terminal tool and report the kernel version"
Each run of this task needs two model calls: one to decide on the tool call, one to read the result and answer. We ran it cold and warm for each model and read the prefill and decode rates from the Ollama log (journalctl -u ollama).
| Model in Ollama | Cold run | Warm run | TTFT cold / warm | Prefill | Decode | Answer |
|---|---|---|---|---|---|---|
| qwen3:4b-instruct-2507-q4_K_M | 2m36s | 20.1s | 124.1s / 0.3s | 35.19 tok/s | 3.44 tok/s | The kernel version is 7.0.0-1017-gcp. |
| llama3.2:3b | 1m16s | 56.5s | 56.2s / 44.0s | 54.48 tok/s | 6.22 tok/s | Linux 7.0.0-1017-gcp |
| qwen3:1.7b | failed in 1.5s | failed | n/a | n/a | n/a | Refused: 40,960 token window |
One-shot hermes -z does not stream, so we could not time the first token from the client. The TTFT column is the prompt eval time of each run’s first model call in the Ollama log, which is the prefill time before the first token at the 64K window (cold runs also pay model load time on top). Cold Qwen sat for 124.1 seconds prefilling 4,367 tokens, and warm it needed 0.3 seconds because the cache held everything but one token. Llama’s cold first call took 56.2 seconds for 3,062 tokens, and warm it still took 44.0 seconds for 2,213 tokens.
Trimming the toolsets cut the cold Qwen run from almost nine minutes to 2m36s, and the request shrank to 4,367 tokens on the first call. Llama 3.2 is faster cold, but its warm run barely improved: Ollama re-evaluated 2,213 prompt tokens on the warm run instead of hitting the cache, and every run opened with an HTTP 400 because Hermes asked it to think. We would stay with the Qwen instruct build for an agent that answers repeatedly.
Qwen’s 1.7B model never got a request out. Hermes checks the context the server reports and stops before calling it, which is the right behavior, but the CLI only says “Something went wrong” and the real reason sits in the error log.

Run ollama show <model> | grep -i context before you pull anything for Hermes. If the native context is under 64K, no Ollama setting will rescue it, because raising the window past what the model was trained on does not make it usable.
Use vLLM as a second backend
Hermes does not care what serves the model as long as it speaks the OpenAI chat API, so vLLM’s CPU build drops into the same base_url slot. It is heavier than Ollama here, but it’s the backend you’d take to a bigger box. We ran the official CPU image on the same host, using the Docker install from earlier.
Stop Ollama’s model first (ollama stop qwen3:4b-instruct-2507-q4_K_M) so the two runtimes do not fight over RAM. The container below serves the unquantized Qwen3 4B instruct model with the same 64K window, a 12 GiB KV cache and the hermes tool-call parser that Qwen models use in vLLM:
export VLLM_CPU_TAG="v0.33.0-x86_64" # pin: tested CPU image #https://github.com/vllm-project/vllm/releases
docker run -d --name vllm-cpu --network host --shm-size=4g \
-e VLLM_CPU_KVCACHE_SPACE=12 \
-v $HOME/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai-cpu:${VLLM_CPU_TAG} \
--model Qwen/Qwen3-4B-Instruct-2507 --served-model-name qwen3-4b-instruct \
--host 127.0.0.1 --port 8000 --max-model-len 65536 \
--enable-auto-tool-choice --tool-call-parser hermes
Follow docker logs -f vllm-cpu until the server line appears. Ours took about nine minutes including the model download, and most of that was engine warm-up and compilation, not the weights. The repeated “No available shared memory broadcast block” lines during warm-up are noise; the KV cache line is the one to read, since it tells you how many full-length requests fit. The vLLM CPU docs now call VLLM_CPU_KVCACHE_SPACE a legacy setting in favour of --kv-cache-memory-bytes; it still took effect in our run, as the log shows:
(Worker pid=122) INFO 11-12 10:16:00 [model_runner.py:428] Model loading took 10.76 GiB memory and 66.364509 seconds
(Worker pid=122) INFO 11-12 10:20:55 [cpu_worker.py:261] Explicitly set (12.0/31.34) GiB for KV cache on node 0.
(EngineCore pid=90) INFO 11-12 10:20:55 [kv_cache_utils.py:1147] CPU KV cache size: 87,296 tokens, Maximum concurrency for 65,536 tokens per request: 1.33x
(EngineCore pid=90) INFO 11-12 10:22:50 [core.py:412] init engine (profile, create kv cache, warmup model) took 410.33 s
(APIServer pid=1) INFO 11-12 10:22:52 [parser_manager.py:36] "auto" tool choice has been enabled.
(APIServer pid=1) INFO 11-12 10:22:53 [entry.py:139] Starting vLLM server on http://127.0.0.1:8000
Once vLLM is listening on port 8000, repoint Hermes. Only three keys change, since the provider type and the 64K context length stay exactly as they were for Ollama:
hermes config set model.base_url http://127.0.0.1:8000/v1
hermes config set model.default qwen3-4b-instruct
hermes config set model.api_key vllm-local
Our uname task came back correct from vLLM, at 7m40s cold and 30.5s warm. Both numbers are slower than Ollama, which makes sense: vLLM ran the unquantized weights while Ollama ran a Q4 quantization, so treat this as proof the wiring works, not as a runtime shootout. For time to first token, vLLM’s /metrics counted four observations summing to 465.0 seconds across both runs. The histogram does not split them per request, so we have no exact cold figure, but the warm run took 30.5 seconds end to end, which puts at least 434 seconds of that on the cold run’s two calls.

One thing to know before you trust the usage file: Hermes reported cache_read_tokens: 0 for both vLLM runs, yet vLLM’s /metrics showed 13,312 of 17,964 prompt tokens served from its prefix cache. vLLM leaves cached counts out of its usage block by default; the --enable-prompt-tokens-details server flag adds them, though we did not test whether Hermes then reads them. For the full production setup, see our vLLM install guide, and Ollama vs vLLM vs llama.cpp covers when each runtime wins.
We pointed Hermes back at Ollama for the rest of the build and stopped the vLLM container, because its 23 GiB of weights and cache plus Ollama’s 12 GB model will not fit together on a 31 GB host:
docker stop vllm-cpu
hermes config set model.base_url http://127.0.0.1:11434/v1
hermes config set model.default qwen3:4b-instruct-2507-q4_K_M
hermes config set model.api_key ollama
Run hermes config get model once more and confirm the base URL ends in 11434 before you move on to the API server.
Enable the Hermes API server on port 8642
Now we switch to the other endpoint: Hermes as a server. The API server is part of the Hermes gateway, and the order of the wiring matters because the gateway reads its settings only when it starts:
- Write the enable flag and a bearer key into
~/.hermes/.env. - Cut the
api_serverplatform down to the terminal and file toolsets. - Install and start the gateway as a user service.
- Test health, auth and the model list with curl.
The key is required even on loopback, so generate a random one and keep it in your shell for the curl tests below. Anyone holding it can run commands on this host through the agent, so treat it like an SSH key.
API_SERVER_KEY=$(openssl rand -hex 24)
echo "API_SERVER_ENABLED=true" >> ~/.hermes/.env
echo "API_SERVER_KEY=$API_SERVER_KEY" >> ~/.hermes/.env
Requests that come in over the API use the api_server platform’s toolsets, not the ones you passed to the CLI. Disable everything except terminal and file so remote callers get the same trimmed, faster agent:
hermes tools disable --platform api_server web browser code_execution vision image_gen skills todo memory session_search connections delegation cronjob
hermes tools list --platform api_server | grep enabled | grep -v disabled
Two toolsets should remain. Everything else is gone from API requests, which keeps each call close to the 4,400 tokens we measured with the trimmed CLI runs:
✓ enabled terminal 💻 Terminal & Processes
✓ enabled file 📁 File Operations
Install the gateway as a user service. The installer also enables linger for your account, so the API keeps running after you log out of SSH instead of dying with the session:
hermes gateway install
hermes gateway status
Look for an active unit. The gateway holds no model, so its memory stays small next to Ollama, and restarting it later costs seconds rather than another cold prefill:
● hermes-gateway.service - Hermes Agent Gateway - Messaging Platform Integration
Loaded: loaded (/home/eugene/.config/systemd/user/hermes-gateway.service; enabled; preset: enabled)
Active: active (running) since Thu 2026-11-12 10:10:37 UTC; 9s ago
Memory: 232.4M (peak: 232.7M)
Test the three things that matter: the port, the unauthenticated health check, and that the model list really demands the key.
ss -ltn | grep 8642
curl -s http://127.0.0.1:8642/health
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:8642/v1/models
curl -s http://127.0.0.1:8642/v1/models -H "Authorization: Bearer $API_SERVER_KEY" | jq
The server listens on loopback only, returns 401 without the key, and exposes one model called hermes-agent. If the port line is empty, check that both .env lines are in place, then run hermes gateway restart so the gateway reads them again:

Behind the hermes-agent name there is no single model. It’s the whole agent loop, so every chat request to it can run tools on this host before it answers. Send one through curl to see the round trip end to end:
time curl -s http://127.0.0.1:8642/v1/chat/completions -H "Authorization: Bearer $API_SERVER_KEY" -H "Content-Type: application/json" -d '{"model": "hermes-agent", "messages": [{"role": "user", "content": "How much free disk space is on / ? Use the terminal tool."}]}' | jq
After 2m39s the answer arrived as a normal chat completion. Any OpenAI client library can read this shape, which is the whole point of the endpoint:
{
"id": "chatcmpl-6a16e11a31e047a0856085b5918a7",
"object": "chat.completion",
"created": 1794478268,
"model": "hermes-agent",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "46782152 bytes, or 46.78 MB, of free disk space on /."
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 9262,
"completion_tokens": 57,
"total_tokens": 9319
}
}
The wiring worked, but the answer is wrong. The model ran df /, which prints 1K blocks, and read 46,781,724 blocks as bytes. df -h / on the same box shows 45G free. A 4B model on CPU will run your tools correctly and still misread the output, so check anything you act on. For scripted callers, ask for raw command output in the prompt and parse it yourself, rather than trusting the model’s arithmetic.
Connect Open WebUI to Hermes
Open WebUI is the client side of the second arrow. We point it at Hermes, not at Ollama, so every chat goes through the agent and its tools. The Hermes docs recommend this setup in their Open WebUI page. We turned off Open WebUI’s own Ollama connection so nobody bypasses the agent by accident. The key variable only lives in the shell where you created it, so reload it from ~/.hermes/.env first in case you reconnected since; an empty key here means a 401 later:
API_SERVER_KEY=$(grep ^API_SERVER_KEY= ~/.hermes/.env | cut -d= -f2)
Then start the container, bound to loopback on port 3000:
export OWUI_TAG="v0.13.3" # pin: tested release #https://github.com/open-webui/open-webui/releases
docker run -d --name open-webui -p 127.0.0.1:3000:8080 \
-e OPENAI_API_BASE_URL=http://host.docker.internal:8642/v1 \
-e OPENAI_API_KEY="$API_SERVER_KEY" \
-e ENABLE_OLLAMA_API=false \
--add-host=host.docker.internal:host-gateway \
-v open-webui:/app/backend/data \
ghcr.io/open-webui/open-webui:${OWUI_TAG}
Before opening the browser, test the hop from inside the container. With Hermes still bound to 127.0.0.1, curl exits with code 7 (connection refused), because the container’s host.docker.internal resolves to the docker0 bridge, not to the host’s loopback.
docker exec open-webui curl -s -m 5 http://host.docker.internal:8642/health; echo "exit=$?"
Bind the API server to the docker0 address instead. Check the address first, since 172.17.0.1 is only Docker’s default, then add API_SERVER_HOST and restart the gateway:
ip -4 addr show docker0 | grep inet
echo API_SERVER_HOST=172.17.0.1 >> ~/.hermes/.env
hermes gateway restart
docker exec open-webui curl -s -m 5 http://host.docker.internal:8642/health
Now the container reaches Hermes, and ss shows the listener on 172.17.0.1:8642 instead of loopback. The gateway log also carries a warning worth reading before you move on:

Hermes flags that it is network-accessible while its terminal backend is local, meaning any caller with the key runs commands as your user. The docker0 address is reachable only from this host and its containers, so the exposure is small, but keep port 8642 firewalled and consider terminal.backend: docker as the log suggests. Never bind it to 0.0.0.0, and remember that every other container on the default bridge can now reach the port too, so the key is the only thing between them and your shell.
The new bind also moves the endpoint for local callers. Hermes no longer listens on 127.0.0.1, so the earlier curl tests and any scripts on the host must use http://172.17.0.1:8642 from now on. We did not test a reboot: the gateway has to start after Docker has created docker0 or the bind fails, so run hermes gateway restart if 8642 is missing after boot. Neither container has a restart policy either, so add --restart unless-stopped to the docker run lines if they should come back on their own.
Open WebUI itself stays on 127.0.0.1:3000. From your workstation, tunnel to it with ssh -N -L 3000:127.0.0.1:3000 [email protected] and browse to http://localhost:3000. The first account you create becomes the admin. If a team needs it, put it behind a reverse proxy with HTTPS as in our Open WebUI install guide; never expose plain HTTP.
Our connection came from the environment variables in the docker run command, so the UI only needs a check, not a setup:
- Click your avatar, open Admin Settings and go to Connections.
- Find http://host.docker.internal:8642/v1 under the OpenAI API connections, and check the toggle below it reads Ollama API off.
- Click Configure (the gear) to open the connection.

- 1Connections
- 2http://host.docker.internal:8642/v1
- 3Configure
- 4Ollama API off
In the Edit Connection dialog, both ends of the handshake have to match what Hermes holds in ~/.hermes/.env, so check each field in order:
- URL is the Hermes address as the container sees it, ending in
/v1. - Auth: Bearer is the scheme the Hermes API server expects.
- API Key holds the same value as
API_SERVER_KEY. - Click Verify Connection and wait for the “Server connection verified” toast.
- Click Save to close the dialog.
A failed verify here is almost always the key or the bind address, and the troubleshooting section below covers both. Our dialog verified on the first try once the gateway was bound to docker0:

- 1URL
- 2Auth: Bearer
- 3API Key
- 4Verify Connection
- 5Save
Start a new chat, pick hermes-agent in the model selector, and type a task that needs a tool into the Prompt box. We asked it to run uname -r and report the kernel.

- 1hermes-agent
- 2Prompt
Hermes answered with the right kernel. Now the browser can reach the agent, and the agent can reach the model and run tools on the host:

Our first reply took about 260 seconds, though, and the Ollama log explained why. One message produced nine /v1/chat/completions calls, two of them close to three minutes each. Open WebUI generates a title, follow-up suggestions and tags after each reply, and its task model defaults to the current model. Here that is hermes-agent, so each background task became a full agent run competing for the same CPU.
Turn off the background tasks
On a CPU host, switch those three off. In Admin Settings, open Interface and set Title Generation off, Follow Up off and Tags off, then click Save.

- 1Title Generation off
- 2Follow Up off
- 3Tags off
- 4Save
With those off, the same message made two Ollama calls (9.8s and 12.5s) and the reply rendered in 25 seconds. The model was already warm for that second test, so do not read it as a clean tenfold speedup. What really changed is that the agent no longer competes with three extra agent runs of itself, worth seven model calls on this message. Titles and tags would need a separate, cheaper task model, and with the Ollama API switched off and Ollama on loopback this setup deliberately has none, so we left them off.
Troubleshoot the link
Every error below came up in our lab. Most of them are on the Hermes to model hop, and the CLI message often hides the cause, so check ~/.hermes/logs/errors.log early. For the API server side, ~/.hermes/logs/gateway.log shows the bind address and every warning the gateway raised at startup.
provider ‘ollama’ has no endpoint configured
We got this with model.provider set to ollama and no URL anywhere. Hermes refuses to send the request rather than fall back to OpenRouter, which is the safe choice for a local setup.

Switch the provider to custom and set the URL, key and context length exactly as in the wiring section above. The message also offers a per-provider URL under providers.ollama, which we did not test; keeping everything under model means one block holds the whole link.
provider ‘custom’ resolved without credentials
Here is the “custom endpoint not connecting” case. We exported OPENAI_BASE_URL=http://127.0.0.1:11434/v1 with the provider set to custom, and Hermes ignored it. Per the Hermes providers docs, that variable is honored only for the openai-api provider (we only tested the custom half). The docs put the endpoint in model.base_url in config.yaml; we wrote it there, with a placeholder key, using hermes config set.
This conversation has grown too long for the model to read
It shows on the very first message and looks like a history problem, but it’s Ollama serving the default 4,096 window. ollama ps confirms it in the CONTEXT column:
NAME ID SIZE PROCESSOR CONTEXT RUNNER UNTIL
qwen3:4b-instruct-2507-q4_K_M 0edcdef34593 3.2 GB 100% CPU 4096 llamacpp 4 minutes from now
Add the OLLAMA_CONTEXT_LENGTH=65536 drop-in, restart Ollama, and the same column reads 65536. Running /new or /compress won’t help, because the tool schemas alone overflow 4K. Expect the model’s memory to jump from 3.2 GB to about 12 GB once the bigger window loads.
Something went wrong and I couldn’t finish this reply
With qwen3:1.7b this generic message hid a context check. The error log names it, together with the fix for each runtime:
ValueError: Model qwen3:1.7b has a context window of 40,960 tokens, which is below the minimum 64,000 required by Hermes Agent. Your local server is serving a 40,960-token window. Start it with at least 64K context (llama.cpp: -c 64000; vLLM: --max-model-len; Ollama: OLLAMA_CONTEXT_LENGTH=64000 or a Modelfile num_ctx), or set model.ollama_num_ctx in config.yaml to the window it really serves (at least 64K).
For this model the server setting can’t fix it, since 40,960 is its native maximum. Pick a model whose ollama show context length is 64K or more; in our lab the Qwen3 4B instruct build reported 262144 and llama3.2:3b reported 131072, so both clear the bar.
“llama3.2:3b” does not support thinking
Each llama3.2 run started with an HTTP 400 from Ollama, logged in errors.log as “Streaming failed before delivery”. Hermes retried on its own and the task still finished, so the cost is one wasted round trip per run. The Qwen instruct build never triggered it, which is one more reason we kept it as the default model.
Open WebUI shows no models or curl exits with code 7
The container can’t reach Hermes on the host’s loopback. Bind API_SERVER_HOST to the docker0 address, restart the gateway, and test from inside the container with docker exec open-webui curl before touching the UI. A 401 instead means the key in Open WebUI doesn’t match API_SERVER_KEY.
Once both hops answer, the chain holds end to end: Open WebUI to Hermes on 8642, Hermes to Ollama on 11434, and tools on the host. The API server docs cover the rest of the gateway settings, including the concurrent run limit. For a different harness on the same local model setup, our DeepSeek harness guide wires a similar stack, and the Ollama CPU guide for Fedora covers tuning Ollama itself on hosts without a GPU.