AI

Hermes Agent with Ollama on CPU: Local Model and API Server Setup

You have a Linux box with plenty of RAM, no GPU, and you want Hermes Agent running against a model that never leaves the machine. Two endpoints matter here, and most of the confusion we hit came from mixing them up. Hermes calls a model endpoint (Ollama on port 11434, or vLLM on 8000), and Hermes also serves its own OpenAI-compatible API on port 8642 for curl and Open WebUI.

Original content from computingforgeeks.com - post 189138

Below we wire both sides of the Hermes Agent Ollama setup on a CPU-only host and record the timings, because on CPU the numbers decide whether the agent is usable at all. Every step ran on an 8 vCPU, 31 GB cloud VM with no GPU. We confirmed the chain working on Ubuntu 26.04 with Hermes Agent 0.23.1 and Ollama 0.40.16 in November 2026.

Keep the two endpoints apart

Open WebUI (or curl) talks to the Hermes API server, Hermes runs its agent loop and tools, and every model call goes to Ollama, with vLLM as an optional second backend on the same box. The diagram below is the whole wiring, numbered in the order a request travels.

Architecture diagram of Hermes Agent with Ollama on CPU: Open WebUI and curl call the Hermes API server on 8642, which calls Ollama on 11434 or vLLM on 8000

When something breaks, the first question is always which hop failed: the client to Hermes, or Hermes to the model. Each hop has its own port, its own key and its own log, so we test them one at a time in the sections that follow, model side first, then the API server, then the browser.

Check how much RAM the 64K window needs

Hermes Agent refuses any model whose context window is under 64,000 tokens, and that one rule sets the memory budget. A small model at 64K needs far more RAM than its download size suggests, because the KV cache for the whole window is allocated up front. We read these figures on the lab host from ollama ps and the vLLM startup log.

What runsMemory measuredWhat drives it
qwen3:4b-instruct Q4_K_M in Ollama, 4,096 context3.2 GBOllama’s default window, too small for Hermes
qwen3:4b-instruct Q4_K_M in Ollama, 65,536 context12 GBKV cache for the full 64K window
llama3.2:3b in Ollama, 65,536 context10 GBSame window, smaller model
Qwen3-4B-Instruct in vLLM CPU10.76 GiB weights plus 12 GiB KV cacheUnquantized weights, VLLM_CPU_KVCACHE_SPACE=12
Hermes gateway (API server)232 MBPython process, no model inside

Treat our 8 vCPU and 31 GB as the floor for one user, not a production size. Prefill on CPU is compute-bound, so more cores shorten the cold runs, and each extra concurrent session needs its own KV space: vLLM reported room for just 1.33 full 64K requests at once with 12 GiB of cache. If you are picking between models first, our open source LLM comparison covers the families. Our Xeon reported avx512f in lscpu; check the instruction sets your CPU offers before you try the vLLM backend.

Install Ollama and Hermes Agent

Start on the model side. Ollama’s install script sets up a systemd service that listens on 127.0.0.1:11434, which is exactly where Hermes will look for it later. Loopback is the right place for it here, because only Hermes on the same host needs to reach the model, and nothing else should.

curl -fsSL https://ollama.com/install.sh | sh

We pulled the Qwen3 4B instruct build at Q4_K_M as the main model, plus two smaller ones to compare later. The instruct build matters on a CPU host, because it answers tool calls directly instead of spending slow decode time on a thinking block first.

ollama pull qwen3:4b-instruct-2507-q4_K_M
ollama pull llama3.2:3b
ollama pull qwen3:1.7b

Now the agent side. The Hermes installer runs as your normal user and puts everything under ~/.hermes, including config.yaml, the .env file and the logs we read later. On a headless server we skip the browser and computer-use extras, since neither does anything without a desktop and both add install time.

curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash -s -- --non-interactive --skip-browser --skip-computer-use
source ~/.bashrc

Confirm both binaries answer before wiring anything, since a missing PATH entry is easier to spot now than after a failed agent run.

hermes --version
ollama --version

Our lab returned the Hermes release line, install details and the Ollama version:

Hermes Agent v0.23.1 (2026.9.24) · stable · local 3b4215785882
Install directory: /home/eugene/.hermes/hermes-agent
Install method: git
Python: 3.14.9
OpenAI SDK: 2.54.0
Up to date
ollama version is 0.40.16

For the day-to-day Ollama commands (pull, ps, show, rm) keep the Ollama commands cheat sheet open, and use the Ollama models cheat sheet when you pick a different model tag. Both sides are installed now, but they do not know about each other yet.

Open WebUI and the optional vLLM backend both run as containers later on, so install Docker now from the Ubuntu archive and add your user to the docker group. The newgrp line applies the group in the current shell without logging out:

sudo apt-get install -y docker.io
sudo usermod -aG docker $USER
newgrp docker

The Ubuntu package was enough for both containers in this guide, so there is no need to add Docker’s own repository for a lab like this one.

Set Ollama to a 64K context window

Everyone skips this step, and Hermes punishes it. On a host with no GPU, Ollama serves every model at 4,096 tokens unless told otherwise, even when the model itself supports 262,144. Hermes’ system prompt and tool schemas alone are bigger than that, so the very first message fails with a “conversation has grown too long” error. In our lab, ollama ps showed the Qwen model at a 4096 CONTEXT and 3.2 GB before the change, and at 65536 and 12 GB after it.

The fix belongs on the Ollama side: set OLLAMA_CONTEXT_LENGTH=65536 in a systemd drop-in for ollama.service, using the drop-in in the Ollama section of our Hermes Agent cheat sheet. We tested only its first Environment= line, the context length. The other three (flash attention, a q8_0 KV cache, one loaded model) are optional and change the memory and timing numbers in this article. We picked 65536 over the bare 64000 minimum so the window is a power of two with a little margin. It applies to every model Ollama loads, so each one reserves RAM for the full window. Run daemon-reload, restart Ollama, then confirm the variable reached the service, because a model that was already loaded keeps its old window until the restart unloads it:

systemctl show ollama -p Environment

With only the context line added, the variable shows up in the environment line like this. If it is missing, the drop-in file name or the [Service] header is wrong, and Ollama keeps serving 4K:

Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin OLLAMA_CONTEXT_LENGTH=65536

The cheat sheet covers the rest of the Ollama tuning for Hermes. On a CPU host, 64K is the only setting you can’t skip, and the next section proves it with the first real reply.

Point Hermes at the Ollama endpoint

Now we connect the agent to the model. The source of truth is ~/.hermes/config.yaml, and the custom provider with an explicit base_url is the wiring that worked for us. Picking the ollama provider without a URL, or exporting OPENAI_BASE_URL while the provider is custom, both left Hermes with nothing to call (the errors are in the troubleshooting section). The interactive hermes model command is the other route; per the providers docs it writes the provider, base_url and model to config.yaml (plus an API key for endpoints that need one), but we did not run it in this lab. The hermes config set lines below are the path we tested.

hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:11434/v1
hermes config set model.default qwen3:4b-instruct-2507-q4_K_M
hermes config set model.api_key ollama
hermes config set model.context_length 65536

Ollama ignores the API key, and we set a placeholder so the endpoint carries an explicit credential rather than relying on Hermes’ keyless default. The context_length line tells Hermes what window the server really gives it, and it should match the drop-in. Read the block back:

hermes config get model

All five keys should be present, with the API key masked in the output:

default: qwen3:4b-instruct-2507-q4_K_M
provider: custom
base_url: http://127.0.0.1:11434/v1
api_key: '***'
context_length: 65536

Send a one-shot prompt with time in front, then check what Ollama loaded. Be patient on the first one, because ours took almost nine minutes with no output at all while the CPU worked through the prompt, and the usage numbers after the output explain where that time went.

time hermes -z "Reply with the word ready"
ollama ps

The reply came back, and ollama ps confirms the 64K window and the 12 GB footprint, all of it on CPU:

ready

real	8m51.965s
user	0m0.914s
sys	0m0.093s
NAME                             ID              SIZE     PROCESSOR    CONTEXT    RUNNER      UNTIL
qwen3:4b-instruct-2507-q4_K_M    0edcdef34593    12 GB    100% CPU     65536      llamacpp    4 minutes from now    

Nearly nine minutes for one word looks broken, but it is not. With every default toolset enabled, Hermes sent about 11,600 tokens of system prompt and tool schemas with that two-word prompt, and the CPU had to prefill all of them. The same prompt run straight after took 2.9 seconds, because Ollama reused 11,575 of those tokens from its cache.

Hermes Agent with Ollama on CPU taking 8m52s for the first reply and 2.9s warm, with ollama ps showing 12 GB at 65536 context

Ollama’s prompt cache is the reason the agent stays usable on CPU. Anything that changes the start of the prompt (a new toolset, a different system prompt, another model loaded in between) throws it away, and you pay the full prefill again on the next request, so keep one model loaded and one toolset per platform.

Disable extra toolsets and measure the models

On CPU, the cheapest speedup is fewer tools. Every enabled toolset adds schemas to each request, and a server agent rarely needs a browser, image generation or vision. We limited the CLI runs to the terminal and file toolsets with -t, and used --yolo so the terminal command runs without the approval prompt in one-shot mode.

time hermes -m qwen3:4b-instruct-2507-q4_K_M -t terminal,file --yolo --usage-file usage-qwen4i-cold.json -z "Run uname -r with the terminal tool and report the kernel version"

Each run of this task needs two model calls: one to decide on the tool call, one to read the result and answer. We ran it cold and warm for each model and read the prefill and decode rates from the Ollama log (journalctl -u ollama).

Model in OllamaCold runWarm runTTFT cold / warmPrefillDecodeAnswer
qwen3:4b-instruct-2507-q4_K_M2m36s20.1s124.1s / 0.3s35.19 tok/s3.44 tok/sThe kernel version is 7.0.0-1017-gcp.
llama3.2:3b1m16s56.5s56.2s / 44.0s54.48 tok/s6.22 tok/sLinux 7.0.0-1017-gcp
qwen3:1.7bfailed in 1.5sfailedn/an/an/aRefused: 40,960 token window

One-shot hermes -z does not stream, so we could not time the first token from the client. The TTFT column is the prompt eval time of each run’s first model call in the Ollama log, which is the prefill time before the first token at the 64K window (cold runs also pay model load time on top). Cold Qwen sat for 124.1 seconds prefilling 4,367 tokens, and warm it needed 0.3 seconds because the cache held everything but one token. Llama’s cold first call took 56.2 seconds for 3,062 tokens, and warm it still took 44.0 seconds for 2,213 tokens.

Trimming the toolsets cut the cold Qwen run from almost nine minutes to 2m36s, and the request shrank to 4,367 tokens on the first call. Llama 3.2 is faster cold, but its warm run barely improved: Ollama re-evaluated 2,213 prompt tokens on the warm run instead of hitting the cache, and every run opened with an HTTP 400 because Hermes asked it to think. We would stay with the Qwen instruct build for an agent that answers repeatedly.

Qwen’s 1.7B model never got a request out. Hermes checks the context the server reports and stops before calling it, which is the right behavior, but the CLI only says “Something went wrong” and the real reason sits in the error log.

Hermes Agent refusing qwen3:1.7b because its 40,960 token context window is below the 64,000 token minimum, with ollama show context lengths

Run ollama show <model> | grep -i context before you pull anything for Hermes. If the native context is under 64K, no Ollama setting will rescue it, because raising the window past what the model was trained on does not make it usable.

Use vLLM as a second backend

Hermes does not care what serves the model as long as it speaks the OpenAI chat API, so vLLM’s CPU build drops into the same base_url slot. It is heavier than Ollama here, but it’s the backend you’d take to a bigger box. We ran the official CPU image on the same host, using the Docker install from earlier.

Stop Ollama’s model first (ollama stop qwen3:4b-instruct-2507-q4_K_M) so the two runtimes do not fight over RAM. The container below serves the unquantized Qwen3 4B instruct model with the same 64K window, a 12 GiB KV cache and the hermes tool-call parser that Qwen models use in vLLM:

export VLLM_CPU_TAG="v0.33.0-x86_64" # pin: tested CPU image #https://github.com/vllm-project/vllm/releases
docker run -d --name vllm-cpu --network host --shm-size=4g \
  -e VLLM_CPU_KVCACHE_SPACE=12 \
  -v $HOME/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai-cpu:${VLLM_CPU_TAG} \
  --model Qwen/Qwen3-4B-Instruct-2507 --served-model-name qwen3-4b-instruct \
  --host 127.0.0.1 --port 8000 --max-model-len 65536 \
  --enable-auto-tool-choice --tool-call-parser hermes

Follow docker logs -f vllm-cpu until the server line appears. Ours took about nine minutes including the model download, and most of that was engine warm-up and compilation, not the weights. The repeated “No available shared memory broadcast block” lines during warm-up are noise; the KV cache line is the one to read, since it tells you how many full-length requests fit. The vLLM CPU docs now call VLLM_CPU_KVCACHE_SPACE a legacy setting in favour of --kv-cache-memory-bytes; it still took effect in our run, as the log shows:

(Worker pid=122) INFO 11-12 10:16:00 [model_runner.py:428] Model loading took 10.76 GiB memory and 66.364509 seconds
(Worker pid=122) INFO 11-12 10:20:55 [cpu_worker.py:261] Explicitly set (12.0/31.34) GiB for KV cache on node 0.
(EngineCore pid=90) INFO 11-12 10:20:55 [kv_cache_utils.py:1147] CPU KV cache size: 87,296 tokens, Maximum concurrency for 65,536 tokens per request: 1.33x
(EngineCore pid=90) INFO 11-12 10:22:50 [core.py:412] init engine (profile, create kv cache, warmup model) took 410.33 s
(APIServer pid=1) INFO 11-12 10:22:52 [parser_manager.py:36] "auto" tool choice has been enabled.
(APIServer pid=1) INFO 11-12 10:22:53 [entry.py:139] Starting vLLM server on http://127.0.0.1:8000

Once vLLM is listening on port 8000, repoint Hermes. Only three keys change, since the provider type and the 64K context length stay exactly as they were for Ollama:

hermes config set model.base_url http://127.0.0.1:8000/v1
hermes config set model.default qwen3-4b-instruct
hermes config set model.api_key vllm-local

Our uname task came back correct from vLLM, at 7m40s cold and 30.5s warm. Both numbers are slower than Ollama, which makes sense: vLLM ran the unquantized weights while Ollama ran a Q4 quantization, so treat this as proof the wiring works, not as a runtime shootout. For time to first token, vLLM’s /metrics counted four observations summing to 465.0 seconds across both runs. The histogram does not split them per request, so we have no exact cold figure, but the warm run took 30.5 seconds end to end, which puts at least 434 seconds of that on the cold run’s two calls.

Hermes Agent pointed at vLLM CPU on port 8000 answering the uname task in 7m40s cold and 30.5s warm

One thing to know before you trust the usage file: Hermes reported cache_read_tokens: 0 for both vLLM runs, yet vLLM’s /metrics showed 13,312 of 17,964 prompt tokens served from its prefix cache. vLLM leaves cached counts out of its usage block by default; the --enable-prompt-tokens-details server flag adds them, though we did not test whether Hermes then reads them. For the full production setup, see our vLLM install guide, and Ollama vs vLLM vs llama.cpp covers when each runtime wins.

We pointed Hermes back at Ollama for the rest of the build and stopped the vLLM container, because its 23 GiB of weights and cache plus Ollama’s 12 GB model will not fit together on a 31 GB host:

docker stop vllm-cpu
hermes config set model.base_url http://127.0.0.1:11434/v1
hermes config set model.default qwen3:4b-instruct-2507-q4_K_M
hermes config set model.api_key ollama

Run hermes config get model once more and confirm the base URL ends in 11434 before you move on to the API server.

Enable the Hermes API server on port 8642

Now we switch to the other endpoint: Hermes as a server. The API server is part of the Hermes gateway, and the order of the wiring matters because the gateway reads its settings only when it starts:

  1. Write the enable flag and a bearer key into ~/.hermes/.env.
  2. Cut the api_server platform down to the terminal and file toolsets.
  3. Install and start the gateway as a user service.
  4. Test health, auth and the model list with curl.

The key is required even on loopback, so generate a random one and keep it in your shell for the curl tests below. Anyone holding it can run commands on this host through the agent, so treat it like an SSH key.

API_SERVER_KEY=$(openssl rand -hex 24)
echo "API_SERVER_ENABLED=true" >> ~/.hermes/.env
echo "API_SERVER_KEY=$API_SERVER_KEY" >> ~/.hermes/.env

Requests that come in over the API use the api_server platform’s toolsets, not the ones you passed to the CLI. Disable everything except terminal and file so remote callers get the same trimmed, faster agent:

hermes tools disable --platform api_server web browser code_execution vision image_gen skills todo memory session_search connections delegation cronjob
hermes tools list --platform api_server | grep enabled | grep -v disabled

Two toolsets should remain. Everything else is gone from API requests, which keeps each call close to the 4,400 tokens we measured with the trimmed CLI runs:

  ✓ enabled  terminal  💻 Terminal & Processes
  ✓ enabled  file  📁 File Operations

Install the gateway as a user service. The installer also enables linger for your account, so the API keeps running after you log out of SSH instead of dying with the session:

hermes gateway install
hermes gateway status

Look for an active unit. The gateway holds no model, so its memory stays small next to Ollama, and restarting it later costs seconds rather than another cold prefill:

● hermes-gateway.service - Hermes Agent Gateway - Messaging Platform Integration
     Loaded: loaded (/home/eugene/.config/systemd/user/hermes-gateway.service; enabled; preset: enabled)
     Active: active (running) since Thu 2026-11-12 10:10:37 UTC; 9s ago
     Memory: 232.4M (peak: 232.7M)

Test the three things that matter: the port, the unauthenticated health check, and that the model list really demands the key.

ss -ltn | grep 8642
curl -s http://127.0.0.1:8642/health
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:8642/v1/models
curl -s http://127.0.0.1:8642/v1/models -H "Authorization: Bearer $API_SERVER_KEY" | jq

The server listens on loopback only, returns 401 without the key, and exposes one model called hermes-agent. If the port line is empty, check that both .env lines are in place, then run hermes gateway restart so the gateway reads them again:

Hermes Agent API server on 127.0.0.1:8642 returning health ok, 401 without a key and the hermes-agent model with the bearer key

Behind the hermes-agent name there is no single model. It’s the whole agent loop, so every chat request to it can run tools on this host before it answers. Send one through curl to see the round trip end to end:

time curl -s http://127.0.0.1:8642/v1/chat/completions -H "Authorization: Bearer $API_SERVER_KEY" -H "Content-Type: application/json" -d '{"model": "hermes-agent", "messages": [{"role": "user", "content": "How much free disk space is on / ? Use the terminal tool."}]}' | jq

After 2m39s the answer arrived as a normal chat completion. Any OpenAI client library can read this shape, which is the whole point of the endpoint:

{
  "id": "chatcmpl-6a16e11a31e047a0856085b5918a7",
  "object": "chat.completion",
  "created": 1794478268,
  "model": "hermes-agent",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "46782152 bytes, or 46.78 MB, of free disk space on /."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 9262,
    "completion_tokens": 57,
    "total_tokens": 9319
  }
}

The wiring worked, but the answer is wrong. The model ran df /, which prints 1K blocks, and read 46,781,724 blocks as bytes. df -h / on the same box shows 45G free. A 4B model on CPU will run your tools correctly and still misread the output, so check anything you act on. For scripted callers, ask for raw command output in the prompt and parse it yourself, rather than trusting the model’s arithmetic.

Connect Open WebUI to Hermes

Open WebUI is the client side of the second arrow. We point it at Hermes, not at Ollama, so every chat goes through the agent and its tools. The Hermes docs recommend this setup in their Open WebUI page. We turned off Open WebUI’s own Ollama connection so nobody bypasses the agent by accident. The key variable only lives in the shell where you created it, so reload it from ~/.hermes/.env first in case you reconnected since; an empty key here means a 401 later:

API_SERVER_KEY=$(grep ^API_SERVER_KEY= ~/.hermes/.env | cut -d= -f2)

Then start the container, bound to loopback on port 3000:

export OWUI_TAG="v0.13.3" # pin: tested release #https://github.com/open-webui/open-webui/releases
docker run -d --name open-webui -p 127.0.0.1:3000:8080 \
  -e OPENAI_API_BASE_URL=http://host.docker.internal:8642/v1 \
  -e OPENAI_API_KEY="$API_SERVER_KEY" \
  -e ENABLE_OLLAMA_API=false \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:${OWUI_TAG}

Before opening the browser, test the hop from inside the container. With Hermes still bound to 127.0.0.1, curl exits with code 7 (connection refused), because the container’s host.docker.internal resolves to the docker0 bridge, not to the host’s loopback.

docker exec open-webui curl -s -m 5 http://host.docker.internal:8642/health; echo "exit=$?"

Bind the API server to the docker0 address instead. Check the address first, since 172.17.0.1 is only Docker’s default, then add API_SERVER_HOST and restart the gateway:

ip -4 addr show docker0 | grep inet
echo API_SERVER_HOST=172.17.0.1 >> ~/.hermes/.env
hermes gateway restart
docker exec open-webui curl -s -m 5 http://host.docker.internal:8642/health

Now the container reaches Hermes, and ss shows the listener on 172.17.0.1:8642 instead of loopback. The gateway log also carries a warning worth reading before you move on:

Open WebUI container failing to reach Hermes on 127.0.0.1, then API_SERVER_HOST set to the docker0 address 172.17.0.1 and the health check passing

Hermes flags that it is network-accessible while its terminal backend is local, meaning any caller with the key runs commands as your user. The docker0 address is reachable only from this host and its containers, so the exposure is small, but keep port 8642 firewalled and consider terminal.backend: docker as the log suggests. Never bind it to 0.0.0.0, and remember that every other container on the default bridge can now reach the port too, so the key is the only thing between them and your shell.

The new bind also moves the endpoint for local callers. Hermes no longer listens on 127.0.0.1, so the earlier curl tests and any scripts on the host must use http://172.17.0.1:8642 from now on. We did not test a reboot: the gateway has to start after Docker has created docker0 or the bind fails, so run hermes gateway restart if 8642 is missing after boot. Neither container has a restart policy either, so add --restart unless-stopped to the docker run lines if they should come back on their own.

Open WebUI itself stays on 127.0.0.1:3000. From your workstation, tunnel to it with ssh -N -L 3000:127.0.0.1:3000 [email protected] and browse to http://localhost:3000. The first account you create becomes the admin. If a team needs it, put it behind a reverse proxy with HTTPS as in our Open WebUI install guide; never expose plain HTTP.

Our connection came from the environment variables in the docker run command, so the UI only needs a check, not a setup:

  1. Click your avatar, open Admin Settings and go to Connections.
  2. Find http://host.docker.internal:8642/v1 under the OpenAI API connections, and check the toggle below it reads Ollama API off.
  3. Click Configure (the gear) to open the connection.
Open WebUI Admin Settings Connections page with the Hermes Agent API server URL on port 8642 and the Ollama API switched off
  1. 1Connections
  2. 2http://host.docker.internal:8642/v1
  3. 3Configure
  4. 4Ollama API off

In the Edit Connection dialog, both ends of the handshake have to match what Hermes holds in ~/.hermes/.env, so check each field in order:

  1. URL is the Hermes address as the container sees it, ending in /v1.
  2. Auth: Bearer is the scheme the Hermes API server expects.
  3. API Key holds the same value as API_SERVER_KEY.
  4. Click Verify Connection and wait for the “Server connection verified” toast.
  5. Click Save to close the dialog.

A failed verify here is almost always the key or the bind address, and the troubleshooting section below covers both. Our dialog verified on the first try once the gateway was bound to docker0:

Open WebUI Edit Connection dialog with the Hermes Agent URL, Bearer auth, the redacted API key and Server connection verified
  1. 1URL
  2. 2Auth: Bearer
  3. 3API Key
  4. 4Verify Connection
  5. 5Save

Start a new chat, pick hermes-agent in the model selector, and type a task that needs a tool into the Prompt box. We asked it to run uname -r and report the kernel.

Open WebUI new chat with hermes-agent selected in the model selector and a prompt asking Hermes to run uname -r
  1. 1hermes-agent
  2. 2Prompt

Hermes answered with the right kernel. Now the browser can reach the agent, and the agent can reach the model and run tools on the host:

Open WebUI chat reply from Hermes Agent reporting kernel 7.0.0-1017-gcp after running uname -r on the CPU host

Our first reply took about 260 seconds, though, and the Ollama log explained why. One message produced nine /v1/chat/completions calls, two of them close to three minutes each. Open WebUI generates a title, follow-up suggestions and tags after each reply, and its task model defaults to the current model. Here that is hermes-agent, so each background task became a full agent run competing for the same CPU.

Turn off the background tasks

On a CPU host, switch those three off. In Admin Settings, open Interface and set Title Generation off, Follow Up off and Tags off, then click Save.

Open WebUI Admin Settings Interface with Title Generation, Follow Up Generation and Tags Generation switched off
  1. 1Title Generation off
  2. 2Follow Up off
  3. 3Tags off
  4. 4Save

With those off, the same message made two Ollama calls (9.8s and 12.5s) and the reply rendered in 25 seconds. The model was already warm for that second test, so do not read it as a clean tenfold speedup. What really changed is that the agent no longer competes with three extra agent runs of itself, worth seven model calls on this message. Titles and tags would need a separate, cheaper task model, and with the Ollama API switched off and Ollama on loopback this setup deliberately has none, so we left them off.

Every error below came up in our lab. Most of them are on the Hermes to model hop, and the CLI message often hides the cause, so check ~/.hermes/logs/errors.log early. For the API server side, ~/.hermes/logs/gateway.log shows the bind address and every warning the gateway raised at startup.

provider ‘ollama’ has no endpoint configured

We got this with model.provider set to ollama and no URL anywhere. Hermes refuses to send the request rather than fall back to OpenRouter, which is the safe choice for a local setup.

Hermes Agent error provider ollama has no endpoint configured when model.provider is ollama without a base_url

Switch the provider to custom and set the URL, key and context length exactly as in the wiring section above. The message also offers a per-provider URL under providers.ollama, which we did not test; keeping everything under model means one block holds the whole link.

provider ‘custom’ resolved without credentials

Here is the “custom endpoint not connecting” case. We exported OPENAI_BASE_URL=http://127.0.0.1:11434/v1 with the provider set to custom, and Hermes ignored it. Per the Hermes providers docs, that variable is honored only for the openai-api provider (we only tested the custom half). The docs put the endpoint in model.base_url in config.yaml; we wrote it there, with a placeholder key, using hermes config set.

This conversation has grown too long for the model to read

It shows on the very first message and looks like a history problem, but it’s Ollama serving the default 4,096 window. ollama ps confirms it in the CONTEXT column:

NAME                             ID              SIZE      PROCESSOR    CONTEXT    RUNNER      UNTIL
qwen3:4b-instruct-2507-q4_K_M    0edcdef34593    3.2 GB    100% CPU     4096       llamacpp    4 minutes from now    

Add the OLLAMA_CONTEXT_LENGTH=65536 drop-in, restart Ollama, and the same column reads 65536. Running /new or /compress won’t help, because the tool schemas alone overflow 4K. Expect the model’s memory to jump from 3.2 GB to about 12 GB once the bigger window loads.

Something went wrong and I couldn’t finish this reply

With qwen3:1.7b this generic message hid a context check. The error log names it, together with the fix for each runtime:

ValueError: Model qwen3:1.7b has a context window of 40,960 tokens, which is below the minimum 64,000 required by Hermes Agent.  Your local server is serving a 40,960-token window.  Start it with at least 64K context (llama.cpp: -c 64000; vLLM: --max-model-len; Ollama: OLLAMA_CONTEXT_LENGTH=64000 or a Modelfile num_ctx), or set model.ollama_num_ctx in config.yaml to the window it really serves (at least 64K).

For this model the server setting can’t fix it, since 40,960 is its native maximum. Pick a model whose ollama show context length is 64K or more; in our lab the Qwen3 4B instruct build reported 262144 and llama3.2:3b reported 131072, so both clear the bar.

“llama3.2:3b” does not support thinking

Each llama3.2 run started with an HTTP 400 from Ollama, logged in errors.log as “Streaming failed before delivery”. Hermes retried on its own and the task still finished, so the cost is one wasted round trip per run. The Qwen instruct build never triggered it, which is one more reason we kept it as the default model.

Open WebUI shows no models or curl exits with code 7

The container can’t reach Hermes on the host’s loopback. Bind API_SERVER_HOST to the docker0 address, restart the gateway, and test from inside the container with docker exec open-webui curl before touching the UI. A 401 instead means the key in Open WebUI doesn’t match API_SERVER_KEY.

Once both hops answer, the chain holds end to end: Open WebUI to Hermes on 8642, Hermes to Ollama on 11434, and tools on the host. The API server docs cover the rest of the gateway settings, including the concurrent run limit. For a different harness on the same local model setup, our DeepSeek harness guide wires a similar stack, and the Ollama CPU guide for Fedora covers tuning Ollama itself on hosts without a GPU.

Keep reading

Ollama Models Cheat Sheet: Best Models by VRAM, Tested (PDF) AI Ollama Models Cheat Sheet: Best Models by VRAM, Tested (PDF) Open Source LLM Comparison Table (2026) AI Open Source LLM Comparison Table (2026) Claude Code Cheat Sheet: Commands, Shortcuts and Flags (PDF) AI Claude Code Cheat Sheet: Commands, Shortcuts and Flags (PDF) Dual GPU LLM Build: PCIe x8/x8 vs x16/x4, PSU and Cooling AI Dual GPU LLM Build: PCIe x8/x8 vs x16/x4, PSU and Cooling MCP vs RAG vs Agent Skills: When to Use Each AI MCP vs RAG vs Agent Skills: When to Use Each Run Muse Glimmer 30B Locally: Setup and Real Speed Tested AI Run Muse Glimmer 30B Locally: Setup and Real Speed Tested

Leave a Comment

Press ESC to close