Ubuntu inference snaps turn local AI on Ubuntu 26.04 into one command. Run sudo snap install gemma4 and you get the model, an engine picked for your hardware and an OpenAI-compatible API on localhost. That engine isn’t always the fastest one, though. On our 4 vCPU Intel VM it chose OpenVINO for Gemma 4 at 6.9 tokens per second, while the llama.cpp engine in the same snap did 7.4.
Below, you install the Gemma 4, DeepSeek R1, Qwen VL and Nemotron 3 Nano snaps, read how hardware detection scored each engine, and drive Gemma 4 through the chat CLI, web UI, curl and OpenCode. We measured tokens per second, RAM and disk next to plain Ollama on the same VM and kept every error we hit. Every command below was run on Ubuntu 26.04.1 LTS in November 2026 on CPU only, so the GPU engines were not tested.
Check the prerequisites
Inference snaps run on any Ubuntu with snapd, but the model size decides what the machine needs. A 4B to 8B model quantized to 4 bits needs 4.5 to 5 GB of RAM while it answers, plus room for whatever else you run beside it, and our LLM memory calculator sizes the bigger ones. Our lab VM had this:
- Ubuntu 26.04.1 LTS, kernel 7.0, snapd 2.77
- 4 vCPUs of an Intel CPU exposing AVX2 and AVX-512 (the flags
gemma4 machinereports), no GPU - 6 GB RAM, no swap
- 100 GB disk, which peaked at 55 GB used during this guide
jqfor reading the API’s JSON (sudo apt install -y jq)
Treat 6 GB as the floor for a single 4B model answering curl requests. It was enough for the benchmarks, but not for a coding agent. Once OpenCode added its own memory and a long context, the model weights were pushed out of the page cache and generation fell to a crawl (the troubleshooting section has the numbers). Give your machine 8 GB or more if you plan to point an agent at it. For an NVIDIA, AMD or Intel GPU, install the driver from Canonical’s driver guide first, because the engine is chosen at install time from the hardware the snap can see. If you are still on the installer, our Ubuntu 26.04 LTS install walkthrough gets you to a shell.
Install the Gemma 4 inference snap
Each model is its own snap, named after the model. These are the four we installed:
gemma4: Gemma 4, text, vision and toolsdeepseek-r1: DeepSeek R1 distills, reasoningqwen-vl: Qwen 2.5 VL, text and imagesnemotron-3-nano: NVIDIA Nemotron 3 Nano, reasoning and tools
Even the stable channel carries a 2.0.0-beta build today, so expect the CLI to keep changing under you. Start with Gemma 4.
sudo snap install gemma4
Snapd fetched a 21 MB snap first, and the real weight came after it. It ran the engine selection, then pulled the OpenVINO Model Server runtime (388 MB) and the model in two components of 2.8 and 3.3 GB. On the first lab node, from the first install attempt, it took about 23 minutes from start to finish while the Snap Store delivered 3 to 6 MB/s, so plan for that on a slow connection. Once it finishes, status shows the engine, the services and every endpoint:
gemma4 status
Both services are active, and the model is gemma4-e4b-ov, the 4B “effective” Gemma 4 converted to OpenVINO INT4:
engine: intel-cpu
services:
server: active
server-webui: active
entrypoints:
kserve:
url: http://127.0.0.1:8336/v2
openai:
url: http://127.0.0.1:8336/v3
tensorflow-serving:
url: http://127.0.0.1:8336/v1
webui:
url: http://127.0.0.1:8337
model:
name: gemma4-e4b-ov
Read the openai line carefully. Canonical’s OpenCode integration page and most OpenAI client examples use /v1, but on the OpenVINO engine the OpenAI-compatible API lives at /v3, and /v1 is the TensorFlow Serving API. A client pointed at http://127.0.0.1:8336/v1 gets nowhere until you change the path. The screenshot adds the installed versions to the same check:

Keep an eye on the components[3/19] note in snap list. The snap can carry 19 optional pieces (runtimes and model weights for every engine), and you only downloaded the three your machine needs, which changes once you start switching engines.
See which engine hardware detection picked
An engine is a runtime plus the weights built for it. Every snap ships a manifest per engine that lists the devices it needs, and at install time the snap scores each engine against the devices gemma4 machine reports. Only the CPU architecture, vendor and flags and the PCI devices (GPUs, NPUs) count toward the score. The same command prints RAM and free disk, but the selector in this release ignores both, which is why a 6 GB box can end up with a 23 GB model later in this guide. The engine with the highest score wins. List them:
gemma4 engines
An asterisk marks the active engine. Two engines are compatible with a GPU-less Intel box, one is experimental, and the three GPU engines are ruled out.
ENGINE VENDOR SUMMARY COMPAT
intel-cpu* Canonical Ltd Optimized for Intel CPUs yes
cpu Canonical Ltd CPU-optimized for workstations yes
intel-onemkl Canonical Ltd Intel CPU-optimized for workstations usin… exptl
nvidia-gpu Canonical Ltd CUDA optimized for workstation GPUs no
intel-gpu Canonical Ltd Optimized for Intel GPUs no
amd-gpu Canonical Ltd ROCm-optimized for workstation GPUs no
To see why intel-cpu won, print each engine with gemma4 engine <name>. The intel-cpu manifest requires an amd64 CPU with manufacturer-id: GenuineIntel and runs OpenVINO Model Server; it scored 16. The generic cpu engine runs llama.cpp on any amd64 or arm64 CPU and scored 10, so a more specific match beats a generic one:

A high score means the engine fits your hardware best, not that it runs fastest on it. On an AMD CPU the intel-cpu manifest fails the vendor check and you get llama.cpp. On an Intel CPU you get OpenVINO, and our benchmark further down shows that is a win for DeepSeek R1 but a small loss for Gemma 4. The same output also lists the models each engine can serve, which is where the bigger gemma4-12b and gemma4-26b-a4b options appear (llama.cpp engines only).
Chat with the model from the terminal
Every inference snap ships a text-only chat client that talks to its own API. It is the quickest way to confirm the model answers before you wire anything else to it:
gemma4 chat
You get a prompt as soon as the server answers. Our session, with the spinner frames removed, went like this.

Note what happened to /exit. The client has no slash commands, so it sent /exit to the model as a prompt and Gemma politely said goodbye. Press Ctrl+C to leave. Your first prompt after the service starts also takes longer, because that request pages the model weights in from disk. Our cold first answers waited 22 to 24 seconds for the first token, and later ones under a second. On the llama.cpp engines the wait comes back after 10 idle minutes, because their manifests set sleep-idle-seconds: 600 and the model is unloaded.
Call the OpenAI-compatible API with curl
Editors, agents and scripts all go through the API, so test it directly. Ask the server which model ID it serves first, because some servers reject a request whose model field does not match.
curl -s http://127.0.0.1:8336/v3/models | jq
Your model ID is the same name that status printed:
{
"data": [
{
"id": "gemma4-e4b-ov",
"object": "model",
"created": 1793653845,
"owned_by": "OVMS"
}
],
"object": "list"
}
If data comes back as an empty list, the model is still loading. A chat request sent in that window fails with Mediapipe graph definition with requested name is not found, which is OpenVINO’s way of saying “not ready yet”, so wait a few seconds and retry. Then send a chat completion and time it:
time curl -s http://127.0.0.1:8336/v3/chat/completions -H 'Content-Type: application/json' -d '{"model": "gemma4-e4b-ov", "messages": [{"role": "user", "content": "Give me one ss command that lists listening TCP ports with the owning process."}], "max_tokens": 120}' | jq
Gemma answered with sudo ss -tulnp and started explaining the flags until it hit the 120-token cap, which is what "finish_reason": "length" means. The 30-second wall time includes loading the model, since this was the first request after a restart:

OpenVINO returns no timings, so the usage block is the only speed hint you get here. For real tokens per second, use the benchmark script later in this guide. Your API also listens on 127.0.0.1 only until you change it, which we cover once all the snaps are installed.
Use the built-in web UI
Each snap also runs a small chat page, on port 8337 for Gemma 4. On a desktop install, open http://127.0.0.1:8337 in your browser. On a server, the obvious SSH tunnel for port 8337 alone gives you this error page.

Your browser loads the page, but its JavaScript calls the API at 127.0.0.1:8336, so the API port has to be reachable on your workstation under the same number. Forward both ports with their original numbers.
ssh -N -L 8336:127.0.0.1:8336 -L 8337:127.0.0.1:8337 [email protected]
With both ports forwarded the page connects. The two controls worth knowing are easy to miss:

- 1Settings toggle
- 2Engine the snap picked
- 3Thinking switch
- The gear icon top right is the Settings toggle. It opens a panel with the engine, the API endpoint, the model and its capabilities.
- The Engine the snap picked shows as
intel-cpuhere, the same valuegemma4 statusprints. - The Thinking switch sits next to Attach. It is on by default, which makes every answer slower on a CPU.
Turn thinking off for quick questions. With it off, Gemma answered a one-line bash question with a single correct command in a few seconds:

Use Attach to send images, since Gemma 4 is a vision model. The page is handy for a quick test from a desktop, and it has no logins or user accounts, so for daily or shared use you’ll want Open WebUI or another full client.
Switch engines and models
Two commands override what the snap chose: use-engine swaps the runtime and use-model swaps the weights within an engine. Both change system state, so both need root, and running them without sudo stops with a terse error:
Error: permission denied, try again with sudo
If you want the OpenVINO numbers for your own CPU, run the benchmark script from the tokens per second section before this step, because the OpenVINO endpoint and model ID go away once you switch. Then move Gemma 4 from OpenVINO to llama.cpp. Without --assume-yes the command asks before it downloads anything:
sudo gemma4 use-engine cpu --assume-yes
You see the components it needs, then the install and the restart. This one took six minutes for 4.9 GB, and status afterwards shows a different API path.

That moved endpoint is the part to act on (and the moment to rerun the benchmark if you want the llama.cpp E4B number). The llama.cpp engine serves the OpenAI API at http://127.0.0.1:8336/v1 and the model ID is now gemma4-e4b, so every client configured for /v3 and gemma4-e4b-ov breaks until you update it. Smaller or larger weights for the same engine are one more command. We dropped to the 2B model to make room for an agent on the 6 GB VM:
sudo gemma4 use-model gemma4-e2b --assume-yes
Expect 3.2 GB and about seven minutes for that one. Neither command deletes the weights it replaced: after both switches snap list showed components[8/19] for Gemma 4 and the snap held 15 GB of models, which is what prune-cache is for (covered in the removal section). To go back to what the hardware check prefers, run sudo gemma4 use-engine --auto.
Point OpenCode at the local API
Any editor or agent that speaks the OpenAI API can use an inference snap as its model provider, with no API key. We used OpenCode because Canonical documents it for inference snaps and it ships as a snap on Ubuntu. Our OpenCode install guide covers other distros:
sudo snap install opencode --classic
You get OpenCode 2.2.0 from the 2/stable channel. Canonical’s integration page gives a provider block with "api": "openai", and on this OpenCode release that config fails before a single request reaches the snap:
Error: Unsupported package for gemma4/gemma4-e4b:
OpenCode loads custom providers through an AI SDK package named in an npm key, and the one for OpenAI-compatible servers is @ai-sdk/openai-compatible, as the OpenCode provider docs show for llama.cpp. Open your config file.
vim ~/.config/opencode/opencode.jsonc
Paste the provider block that worked for us. It points at the llama.cpp engine’s /v1 endpoint and the 2B model from the previous section. On the default OpenVINO engine the values would be http://127.0.0.1:8336/v3 and gemma4-e4b-ov, but we only ran OpenCode against the llama.cpp engine:
{
"$schema": "https://opencode.ai/config.json",
"provider": {
"gemma4": {
"npm": "@ai-sdk/openai-compatible",
"name": "Gemma 4 Inference Snap",
"options": {
"baseURL": "http://127.0.0.1:8336/v1"
},
"models": {
"gemma4-e2b": {}
}
}
}
}
Give the agent a small job and run it non-interactively so you can time it. The --auto flag approves the file write and the shell command the agent asks for:
mkdir -p ~/demo && cd ~/demo
opencode run -m gemma4/gemma4-e2b --auto "Create diskcheck.sh in this directory: a bash script that prints the usage percent of / and exits 1 when it is above 90 percent. Make it executable."
OpenCode called its write tool, ran chmod and then printed the script it had written. The first lines of its output:
> build · gemma4-e2b
← Write diskcheck.sh
Created file successfully: diskcheck.sh
> build · gemma4-e2b
$ chmod +x diskcheck.sh
(no output)
> build · gemma4-e2b
The script `diskcheck.sh` has been created in the current directory, and it has been made executable.
Run the result to make sure the agent did what it claimed.
./diskcheck.sh; echo "exit=$?"
On a disk that was 34% full, the script parsed df correctly and exited 0.
Root partition usage is 34% (OK)
exit=0
It worked, but it took 606 seconds for an 18-line script. Most of that went on reading the prompt, not writing code. OpenCode sends about 4,900 tokens of system prompt and tool definitions with the first request, and the snap’s llama.cpp server read them at 29 tokens per second, so you wait almost three minutes before the agent produces anything. That’s fine for a job you start and walk away from. For interactive coding on a CPU, a hosted model or a GPU engine is the realistic choice, and our OpenCode vs Claude Code comparison covers that trade.
Install DeepSeek R1, Qwen VL and Nemotron 3 Nano
Every other model installs the same way, gets its own pair of ports and runs its own engine selection. Install DeepSeek R1 and Qwen VL next. Every snap starts its server as soon as it is installed, so on a 6 GB machine stop the ones you aren’t using right away with sudo snap stop <snap>. The OpenVINO servers keep their model in memory even when idle.
sudo snap install deepseek-r1
sudo snap install qwen-vl
DeepSeek R1 also picked intel-cpu and serves deepseek-r1-7b-ov, the 7B Qwen distill in OpenVINO INT4, on port 8324 with the web UI on 8325. It is a reasoning model, so expect a long thinking pass before the answer. Qwen VL took 8 minutes 50 seconds and 2.9 GB, picked intel-cpu as well, and serves qwen2.5-vl-3b-ov on port 8326. Its snap follows a 2.5/stable track rather than latest, and its engine list adds an intel-npu engine published by Intel for Core Ultra machines. Check the ports any time with <snap> status:
qwen-vl status
A client needs two values from it, the model name and the /v3 path.
engine: intel-cpu
services:
server: active
server-webui: active
entrypoints:
kserve:
url: http://127.0.0.1:8326/v2
openai:
url: http://127.0.0.1:8326/v3
tensorflow-serving:
url: http://127.0.0.1:8326/v1
webui:
url: http://127.0.0.1:8327
model:
name: qwen2.5-vl-3b-ov
Qwen VL is the one to try with images. The API takes the standard OpenAI message format with a base64 data URL, which is too long to type, so we built the request body with a two-line script. Create it:
vim ~/vl.sh
Our version reads ~/status.jpg (an 800-pixel copy of the gemma4 status screenshot above) and writes ~/vl-request.json. Point it at any JPEG you have:
IMG=$(base64 -w0 ~/status.jpg)
printf '{"model": "qwen2.5-vl-3b-ov", "max_tokens": 120, "messages": [{"role": "user", "content": [{"type": "text", "text": "Which engine and which model name does this terminal show?"}, {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,%s"}}]}]}' "$IMG" > ~/vl-request.json
Build the request, then post it from your home directory:
bash ~/vl.sh && cd ~
time curl -s http://127.0.0.1:8326/v3/chat/completions -H "Content-Type: application/json" -d @vl-request.json | jq ".choices[0].message.content, .usage"
Qwen read both values off the image correctly. The first request took 75 seconds because it loaded the model and encoded 642 prompt tokens of image, and the same request a second time took 29 seconds:

Half a minute per image is slow for batch work but fine for the occasional screenshot or scanned receipt, and nothing leaves the machine.
Nemotron 3 Nano is the one to read up on before you install it. The snap’s CPU engine offers a dense 4B model and the 30B-A3B mixture-of-experts model, and the install hook picks the 30B one, six weight components totalling 23.3 GB on disk, without looking at how much RAM the machine has. Our 6 GB VM got all of it. Pinning the 4B component on the install line did not change that:
sudo snap install nemotron-3-nano+llamacpp+model-4b-q4-k-m-gguf
Snapd installed the 4B component as asked, then the install hook ran its automatic selection and queued all six 30B parts anyway. The download log of that install change shows it. Our change was number 39; find yours in the first column of snap changes:
snap tasks 39 | grep Download
The 4B model finished at 22:45, and the next 28 minutes went on weights your 6 GB machine cannot hold in RAM:
Done today at 22:42 UTC today at 22:42 UTC Download snap "nemotron-3-nano" (197) from channel "stable"
Done today at 22:42 UTC today at 22:42 UTC Download component "llamacpp" (54)
Done today at 22:42 UTC today at 22:45 UTC Download component "model-4b-q4-k-m-gguf" (12)
Done today at 22:45 UTC today at 22:50 UTC Download component "model-30b-a3b-q4-k-m-gguf-1-of-6" (196)
Done today at 22:45 UTC today at 22:55 UTC Download component "model-30b-a3b-q4-k-m-gguf-2-of-6" (196)
Done today at 22:45 UTC today at 23:00 UTC Download component "model-30b-a3b-q4-k-m-gguf-3-of-6" (196)
Done today at 22:45 UTC today at 23:05 UTC Download component "model-30b-a3b-q4-k-m-gguf-4-of-6" (196)
Done today at 22:45 UTC today at 23:11 UTC Download component "model-30b-a3b-q4-k-m-gguf-5-of-6" (196)
Done today at 22:45 UTC today at 23:13 UTC Download component "model-30b-a3b-q4-k-m-gguf-6-of-6" (196)
Afterwards nemotron-3-nano status reported nemotron-3-nano-30b-a3b on the cpu engine, with the API on port 8330. We didn’t send it a prompt, because 23 GB of weights on a 6 GB machine can only thrash. Switching to the 4B model took two seconds, since its component was already on disk:
sudo nemotron-3-nano use-model nemotron-3-nano-4b --assume-yes
On a small machine, plan for the full download anyway: let the install finish, switch with use-model, then run sudo nemotron-3-nano prune-cache to drop the 30B components. The disk space itself only comes back after the snapd cache cleanup in the removal section. Once its context was capped (see the troubleshooting entry on slow generation), the 4B model was the fastest thing we ran, at 9.3 to 9.4 tokens per second.
Open the API to other machines
Every inference snap binds its API to 127.0.0.1 until you change http.host. Once you do, anything that can reach the port can use the model without a key, so only do it on a network you control or behind a firewall rule. Every snap takes the same keys; we opened Qwen VL:
sudo qwen-vl set http.host=0.0.0.0 --assume-yes
sudo snap restart qwen-vl
Confirm the new bind address. The web UI on 8327 stays on localhost because it has its own webui.http.host key.
ss -ltn | grep -E ":832[67]"
The API port now listens on all interfaces:
LISTEN 0 4096 0.0.0.0:8326 0.0.0.0:*
LISTEN 0 4096 0.0.0.0:8326 0.0.0.0:*
LISTEN 0 4096 0.0.0.0:8326 0.0.0.0:*
LISTEN 0 4096 0.0.0.0:8326 0.0.0.0:*
LISTEN 0 4096 127.0.0.1:8327 0.0.0.0:*
From another machine, curl http://10.0.1.50:8326/v3/models returned the qwen2.5-vl-3b-ov model list. If UFW is active, allow the port from your subnet only, for example sudo ufw allow from 10.0.1.0/24 to any port 8326 proto tcp. Use the snap’s own port in that rule, not the 11434 you may remember from Ollama.
Measure CPU tokens per second
Neither chat client prints speed, and the OpenVINO server does not return timings in its responses either, so we measured from the client side with a short Python script that streams a completion and times it. It works against any OpenAI-compatible endpoint, including Ollama’s, which is what makes the comparison fair. Create it in your home directory.
vim ~/bench.py
Paste the script. It sends the same prompt three times with temperature 0 and a 256-token cap, and reports time to first token and the decode rate:
import json, sys, time, urllib.request
base, model = sys.argv[1], sys.argv[2]
runs = int(sys.argv[3]) if len(sys.argv) > 3 else 3
prompt = "Explain in about 200 words how a Linux package manager resolves dependencies."
for i in range(runs):
body = json.dumps({"model": model, "messages": [{"role": "user", "content": prompt}],
"max_tokens": 256, "temperature": 0, "stream": True,
"stream_options": {"include_usage": True}}).encode()
req = urllib.request.Request(base + "/chat/completions", body, {"Content-Type": "application/json"})
t0 = time.time(); first = None; n = 0; usage = None
with urllib.request.urlopen(req, timeout=900) as r:
for line in r:
line = line.decode().strip()
if not line.startswith("data:") or line.endswith("[DONE]"):
continue
d = json.loads(line[5:])
if d.get("usage"):
usage = d["usage"]
for c in d.get("choices", []):
delta = c.get("delta", {})
if delta.get("content") or delta.get("reasoning_content") or delta.get("reasoning"):
if first is None:
first = time.time()
n += 1
t1 = time.time()
ct = usage["completion_tokens"] if usage else n
pt = usage["prompt_tokens"] if usage else None
print(f"run {i+1}: prompt_tokens={pt} completion_tokens={ct} ttft={first-t0:.2f}s "
f"total={t1-t0:.2f}s decode={ (ct-1)/(t1-first):.2f} tok/s")
Pass it the OpenAI endpoint your engine reports in status and the model ID from /models. We ran it on each engine before switching away from it, with only one model loaded at a time (sudo snap stop on the others), because two 5 GB models don’t fit in 6 GB. Start Gemma 4 again if you stopped it, then run the script on the default OpenVINO engine:
sudo snap stop qwen-vl deepseek-r1
sudo snap start gemma4
python3 ~/bench.py http://127.0.0.1:8336/v3 gemma4-e4b-ov 3
If you already switched Gemma 4 to llama.cpp and the 2B model, the same check is python3 ~/bench.py http://127.0.0.1:8336/v1 gemma4-e2b 3, and DeepSeek R1 answers on http://127.0.0.1:8324/v3 as deepseek-r1-7b-ov.
All four runs side by side, the two Gemma 4 engines first and then DeepSeek R1 on the snap and on Ollama:

Warm runs are the numbers to compare, so the table uses the mean of runs 2 and 3 (run 2 alone for the capped Nemotron, which we ran twice). The Nemotron rows come from the same script against port 8330.
| Model and engine | Weights | Decode (tok/s) | First token, warm | Server RSS |
|---|---|---|---|---|
| Gemma 4 E4B, snap intel-cpu (OpenVINO) | INT4 OV, 6.1 GB | 6.9 | 0.8 to 1.1 s | 4.8 GiB |
| Gemma 4 E4B, snap cpu (llama.cpp) | Q4_K_M, 4.5 GB | 7.4 | 0.8 s | 4.8 GiB |
| DeepSeek R1 7B, snap intel-cpu (OpenVINO) | INT4 OV, 4.1 GB | 7.1 | 0.7 to 0.9 s | 4.7 GiB |
| DeepSeek R1 7B, Ollama (llama.cpp) | Q4_K_M, 4.7 GB | 5.5 | 0.5 to 0.6 s | 4.7 GiB |
| Nemotron 3 Nano 4B, snap cpu (llama.cpp), default context | Q4_K_M, 2.7 GB | 8.3 | 1.2 to 1.3 s | 5.1 GiB |
| Nemotron 3 Nano 4B, context capped at 8192 | Q4_K_M, 2.7 GB | 9.3 | 1.0 s | 3.9 GiB |
Two results stand out. For DeepSeek R1 the snap’s OpenVINO build decoded 28% faster than Ollama on the same 7B distill, so on an Intel CPU the snap is the faster way to run that model. For Gemma 4 the picture flips: the llama.cpp engine beat OpenVINO by about 8%, which means the engine score picked the slower option for this model on this CPU. Neither gap changes how the model feels in a chat, but over a long agent session 7 versus 5.5 tokens per second adds up. Cold starts cost the same on both runtimes, about 22 to 24 seconds to load a 4 to 6 GB model from disk. The smallest model was the fastest, as you’d expect, and capping Nemotron’s context added another 12% on top because less memory went to an oversized KV cache.
Compare inference snaps with Ollama on the same VM
Ollama is what most people already run for local AI on Ubuntu 26.04, so we installed it on the same VM and pulled the same DeepSeek R1 7B distill. The steps match our Ollama on Ubuntu 26.04 guide:
curl -fsSL https://ollama.com/install.sh | sh
ollama pull deepseek-r1:7b
Ollama 0.40.13 installed with a warning that no GPU was found, and the 4.7 GB pull took 280 seconds. Ollama also exposes an OpenAI-compatible API, at http://127.0.0.1:11434/v1, so the same bench.py ran against it unchanged. On one machine, the two compare like this:
| DeepSeek R1 7B on 4 vCPU | Inference snap (deepseek-r1) | Ollama 0.40.13 |
|---|---|---|
| Install | sudo snap install deepseek-r1 | install script, then ollama pull |
| Runtime picked on this CPU | OpenVINO Model Server | llama.cpp (llama-server runner) |
| Disk, runtime plus model | 4.6 GB | 6.7 GB (2.3 GB runtime, 4.4 GB model) |
| Decode speed, warm | 7.1 tok/s | 5.5 tok/s |
| Server memory with model loaded | 4.7 GiB RSS | 4.7 GiB RSS (5.1 GB in ollama ps) |
| OpenAI API | :8324/v3 | :11434/v1, plus Ollama’s own API |
| Models per install | one (parallel instances for more) | any model in the library |
| Updates | automatic snap refresh | re-run the install script |
Ollama’s runtime is bigger because /usr/local/lib/ollama carries the CUDA 12 and CUDA 13 libraries (1.3 and 1.1 GB) whether or not the machine has a GPU, while a snap downloads only the engine it selected. Memory is a wash: both servers sat at 4.7 GiB with the model loaded. Speed favours the snap on an Intel CPU, because OpenVINO is tuned for exactly that silicon.
We’d take the inference snap when the box serves one known model to tools on the network and you want it updated by snapd like everything else. Ollama stays the better pick when you switch models often, since every model is one ollama pull away and several can sit on disk behind one API, which a snap can only match with parallel instances. Our Ollama commands cheat sheet lists the model management side, and the Ollama vs vLLM vs llama.cpp comparison covers the runtimes underneath.
Fix common inference snap errors
Every message below came up in our lab, with the cause and the fix that worked.
Mediapipe graph definition with requested name is not found
The OpenVINO engine returns this JSON error when a chat request arrives before the model has loaded, usually in the first seconds after snap start or snap restart. At the same moment /v3/models returns an empty data list. Poll that list and send requests only once your model ID appears in it. On our first lab node that took 8 seconds after a restart.
Error: permission denied, try again with sudo
Commands that change the snap’s state need root, and the read-only ones run as your user. Prefix these with sudo:
use-engineto swap the runtimeuse-modelto swap the weightssetto change ports or the bind addressprune-cacheto delete unused componentsunsetto drop a setting again
Everything else (status, engines, models, get, chat) works without it.
Error: Unsupported package for gemma4/gemma4-e4b
OpenCode 2.2 prints this when the provider block uses "api": "openai", as Canonical’s OpenCode page still shows. Replace that line with "npm": "@ai-sdk/openai-compatible" and keep the rest of the block.
Could not connect to http://127.0.0.1:8336/v3: Failed to fetch
The web UI shows this when the browser can reach port 8337 but not the API port. Over SSH, forward 8336 as well, on the same local port number, because the page has 127.0.0.1:8336 written into it. It also appears briefly while the server is still loading the model.
set http.host never returns
sudo <snap> set key=value asks whether to restart the snap. From a script or a non-interactive SSH session there is nobody to answer, and our call was still hanging after 200 seconds when we killed it. Pass --assume-yes, which answers yes and restarts the snap, or --no-restart and restart it yourself. The restart takes a few seconds, so give it time before checking ss -ltn; if the old address is still listed after that, sudo snap restart <snap> applies it, which is what we ran.
Generation falls below 1 token per second
This is memory, not CPU. We ran Gemma 4 E4B on llama.cpp with OpenCode beside it on the 6 GB VM, and free memory dropped to 179 MB. At the same time vmstat showed 330 MB/s of disk reads, and the server log reported tg = 0.19 t/s against 7.4 in the benchmark. The weights are memory-mapped, so under pressure the kernel drops them from the page cache and reads them again for every token. Nemotron 4B did the same at 0.28 tokens per second, with 1.8 GB/s of reads.
The server log explains where the memory goes. The llama.cpp engines start llama-server with four slots and let it size the context itself, and it does not count your system RAM as a limit. For Nemotron 4B on our 6 GB VM the log read:
common_params_fit_impl: projected to use 20077 MiB of host memory vs. 5920 MiB of total host memory
slot load_model: id 0 | task -1 | new slot, n_ctx = 133888
Gemma 4 got 48,640 tokens per slot on E4B and 131,072 on E2B. Cap the context with the snap’s env. keys, which pass environment variables straight to llama-server (here LLAMA_ARG_CTX_SIZE), then restart:
sudo nemotron-3-nano set env.llama-arg-ctx-size=8192 --assume-yes
sudo snap restart nemotron-3-nano
With the cap the slots came up at n_ctx = 8192, server memory dropped from 5.1 to 3.9 GiB, and the same benchmark ran at 9.3 to 9.4 tokens per second instead of 7.9 to 8.7. Eight thousand tokens is plenty for chat and curl. For an agent like OpenCode we’d set 16384 or more, since its first request alone is about 4,900 tokens and every tool call adds to the history. If memory is still short after that, switch to a smaller model with use-model or stop the snaps you are not using.
Remove an inference snap and reclaim the disk
Disk is where inference snaps surprise people, because switching engines or models keeps the old weights. After our two switches, Gemma 4 held 15 GB in /var/lib/snapd/snaps for a 2B model that needs 3.4 GB. To drop everything the active engine doesn’t use, keep the snap and prune it:
sudo gemma4 prune-cache
It asks for confirmation (there is no --assume-yes flag on this one) and lists every component it removes, here the OpenVINO runtime and both 4B weight sets, 11.2 GB in all. Afterwards snap list shows components[3/19] again. To remove a model completely, remove its snap with --purge so snapd does not keep a snapshot of your data:
sudo snap remove --purge deepseek-r1
Removal took four seconds, and no DeepSeek file was left in /var/lib/snapd/snaps and no snapshot in snap saved.

Neither command gave the space back, though. After pruning Gemma 4 and Nemotron and removing DeepSeek, df still showed 55 GB used, because snapd keeps every downloaded snap and component as a hard link in /var/lib/snapd/cache. Canonical’s CLI reference warns that pruned components are deleted “at a later time”, and on our VM that cache held 45 GB, most of it files no installed snap pointed to anymore. Snapd cleans that cache on its own, but slowly: its classic-system policy keeps up to five unreferenced files and only drops older ones after 30 days, with no size cap, and here the files are several gigabytes each. A cache file with a link count of 1 has no other copy on the system, so deleting those is safe, provided snap changes shows nothing still in progress:
sudo find /var/lib/snapd/cache -type f -links 1 -delete
That took the disk from 55 GB to 19 GB used, and the three remaining snaps restarted and served their models normally:

Budget the disk for this. A model you tried for an afternoon is one command to remove, but the bytes stay until snapd cleans its cache or you do, and at 3 to 23 GB per model it fills a 100 GB disk quickly. Check df after each removal, and run the cache cleanup above when the number doesn’t move.