You can run llama.cpp on Linux without compiling anything. The project publishes a CPU tarball for every build, and on a 4-vCPU VM with no GPU it answered a 1B model prompt at 13 tokens a second minutes after download. Two things still bite. A fresh Ubuntu cloud image is missing one runtime library, and a service started with an empty API key file serves the whole API to anyone who can reach the port.
You’ll install from prebuilt binaries or a source build, run a first prompt with llama-cli, serve an OpenAI-compatible API with llama-server under systemd, and finish on llama-quantize and llama-bench. Keep the llama.cpp cheat sheet open for the common llama.cpp commands and llama-server flags. Every command below was run in November 2026 on Ubuntu 26.04, Ubuntu 24.04 and Rocky Linux 10 against llama.cpp v0.9.0 (prebuilt build b12110), on 4-vCPU cloud VMs with no GPU.
Pick prebuilt binaries or a source build
Take the prebuilt tarball unless you need a backend it doesn’t ship. The CPU tarball carries one shared library per CPU generation, from SSE4.2-era chips to Sapphire Rapids and Zen 4, and loads the matching one at start. On our VM that was libggml-cpu-haswell.so, so a source build no longer buys you a faster CPU binary the way it once did. Build from source when you want a backend the release page lacks, or when you want to read and pin the exact code you run.
Here is how the two routes compared on the same 4-vCPU, 16 GB VMs.
| Prebuilt tarball | Source build | |
|---|---|---|
| Time to a working binary | Under a minute (22 MB download) | 9 min 8 s without tests and examples on Ubuntu 26.04, 11 min 32 s with them |
| Extra packages | libgomp1 on a fresh Ubuntu image | Compiler, CMake, Git, OpenSSL headers |
| Version string | 0.9.0-dev (build 12110, commit cd959bae9) | 0.9.0-dev (build 1, commit cd959ba) |
| Ran on Rocky Linux 10 | Yes, the Ubuntu tarball works on glibc 2.39 | Yes, after an ldconfig entry |
| GPU builds | CUDA 12.8, CUDA 13.4, ROCm, Vulkan, SYCL tarballs | Any backend, with its SDK installed |
One release detail explains the two numbers above. llama.cpp now cuts versioned releases such as v0.9.0 next to its nightly bNNNNN builds, and the GitHub “latest” release is the versioned one. Its only asset is nightly-tag.txt, which names the nightly build the release points at (b12110 here), and that build holds the tarballs. The commands below read that file, so they keep working when the next release lands.
Install the prebuilt llama.cpp binaries
Resolve the build behind the latest release first. You need it because the tarball names carry the build number, not the release version.
LLAMA_BUILD=$(curl -sL https://github.com/ggml-org/llama.cpp/releases/latest/download/nightly-tag.txt)
echo "$LLAMA_BUILD"
It printed the nightly build that the current release points at.
b12110
Download the CPU tarball and unpack it into /opt/llama.cpp. The archive has a top-level folder, so --strip-components=1 puts the binaries straight into the target directory. A profile script then adds that directory to every user’s PATH.
curl -LO "https://github.com/ggml-org/llama.cpp/releases/download/${LLAMA_BUILD}/llama-${LLAMA_BUILD}-bin-ubuntu-x64.tar.gz"
sudo mkdir -p /opt/llama.cpp
sudo tar xzf "llama-${LLAMA_BUILD}-bin-ubuntu-x64.tar.gz" -C /opt/llama.cpp --strip-components=1
echo 'export PATH=/opt/llama.cpp:$PATH' | sudo tee /etc/profile.d/llama-cpp.sh
source /etc/profile.d/llama-cpp.sh
On a fresh Ubuntu 26.04 cloud image the first llama-cli --version failed with libgomp.so.1: cannot open shared object file. The binaries link the GCC OpenMP runtime, and minimal images do not carry it until something like build-essential pulls it in. Install it before you go further.
sudo apt install -y libgomp1
The same tarball ran on Rocky Linux 10 with no extra package, because the Rocky 10 cloud image already ships libgomp. If yours doesn’t, sudo dnf install -y libgomp fixes the same error. The capture below shows the whole sequence on Ubuntu 26.04, error included.

You get four tools you will use in this guide, and every one sits in /opt/llama.cpp after the unpack:
llama-clifor prompts and interactive chatllama-serverfor the OpenAI-compatible API and web UIllama-quantizeto shrink a model filellama-benchfor repeatable speed numbers
Next to them sits a single llama command that wraps the same tools as subcommands. This is what ./llama --help listed from the unpacked tarball directory.
Usage: ./llama <command> [options]
Available commands:
serve HTTP API server
cli Command-line interactive interface
download Download a model
version Show version
licenses Show third-party licenses
help Show available commands
Run './llama help all' to show additional commands.
Run './llama <command> --help' for command-specific usage.
We stick with the separate binaries below because they are what scripts, unit files and older guides name. The llama update subcommand refused to run on the tarball install with “Updates are available only when installed from https://llama.app”, which points at the installer script on llama.app. We did not run that script in this lab, so to upgrade a tarball install you repeat the download above.
Build llama.cpp from source
A source build needs a C++ compiler, CMake, Git and the OpenSSL headers. The OpenSSL part matters because the -hf downloader uses it for HTTPS, and CMake printed OpenSSL found: 3.5.5 on Ubuntu 26.04, 3.0.13 on 24.04 and 3.5.8 on Rocky 10. Rocky’s minimal image also lacks the openssl command itself, which the API key step needs later, so it goes in the same line.
sudo apt update
sudo apt install -y build-essential cmake git libssl-dev
sudo dnf install -y gcc-c++ cmake git openssl-devel openssl
Ask the GitHub API for the latest release tag and clone only that tag. A shallow clone keeps the download small, and it tells you exactly which code you built.
LLAMA_TAG=$(curl -s https://api.github.com/repos/ggml-org/llama.cpp/releases/latest | grep -m1 "\"tag_name\"" | cut -d "\"" -f 4)
echo "$LLAMA_TAG"
git clone --branch "$LLAMA_TAG" --depth 1 https://github.com/ggml-org/llama.cpp
cd llama.cpp
Configure without the test suite and the examples, then compile on every core. The default configure also builds the tests and examples, which you don’t need to run a model. Dropping them cut the Ubuntu 26.04 build from 11 min 32 s to 9 min 8 s, and the same flags finished in 7 min 52 s on Ubuntu 24.04 and Rocky 10.
cmake -B build -DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_EXAMPLES=OFF
cmake --build build --config Release -j"$(nproc)"
The binaries land in build/bin/, and you can run them from there. If you want them on the system path, install them under /usr/local.
sudo cmake --install build --prefix /usr/local
On both distros the installed llama-server then failed with libllama-server-impl.so: cannot open shared object file, because the binaries carry no library path of their own and the loader cache doesn’t know the new libraries yet. On Ubuntu, where CMake puts them in /usr/local/lib, refreshing the cache is enough:
sudo ldconfig
On Rocky Linux 10 the libraries go to /usr/local/lib64, which the loader doesn’t search by default. Add that directory first, then refresh the cache.
echo /usr/local/lib64 | sudo tee /etc/ld.so.conf.d/llama-cpp.conf
sudo ldconfig
After that the tag, the build and the installed binary agree. The compilers were GCC 15.2 on Ubuntu 26.04, GCC 13.3 on 24.04 and GCC 14.3 on Rocky 10, and all three built the same commit.

On Ubuntu 26.04 the same fix brought back version: 0.9.0-dev (build 1, commit cd959ba) after a single sudo ldconfig. The rest of this guide uses the Ubuntu 26.04 VM with the prebuilt binaries in /opt/llama.cpp; with a source build, swap that path for /usr/local/bin.
Run your first prompt with llama-cli
The -hf flag downloads a GGUF model from Hugging Face on first use and loads it from the cache after that. Without a quant suffix it takes Q4_K_M and falls back to the repo’s first file when there is no Q4_K_M. Gemma 3 1B from the ggml-org mirror is ungated, is 769 MB at Q4_K_M and doesn’t print a thinking block, which makes it a good first model on a CPU box. -st answers once and exits, and -n caps the reply length.
llama-cli -hf ggml-org/gemma-3-1b-it-GGUF -st -p "In two sentences, what does a reverse proxy do?" -n 120
Download and answer took 12 seconds of wall time. Watch the last line. It shows 32.2 tokens a second reading the prompt and 13.2 tokens a second writing the answer on two cores.

Look for the model under ~/.cache/huggingface/hub, not the old ~/.cache/llama.cpp that older guides mention. llama.cpp now shares the standard Hugging Face cache, so a model you pulled with other Hugging Face tools is reused instead of downloaded twice. Drop -st to get an interactive chat; /exit or Ctrl+C leaves it.
Serve the OpenAI-compatible API with llama-server
llama-server listens on 127.0.0.1 port 9931 by default. Older guides and scripts use 8080, so pass --port explicitly if a client config expects that. Put the API key in a file with --api-key-file instead of --api-key on the command line, because anything in the arguments shows up in ps, shell history and systemctl status for every user on the box.
mkdir -p ~/.config/llama
openssl rand -hex 32 > ~/.config/llama/api-keys
chmod 600 ~/.config/llama/api-keys
Start the server with the same model and an 8K context. -c 8192 sizes the KV cache, and the log reported n_slots = 4, n_ctx_slot = 8192, so four requests can run at once.
llama-server -hf ggml-org/gemma-3-1b-it-GGUF -c 8192 --host 127.0.0.1 --port 9931 --api-key-file ~/.config/llama/api-keys
From a second terminal, check what the key protects. /health stays open so load balancers and probes work without a key, while /v1/models and a request with the wrong key both get 401.

Now send a chat completion with the key in a Bearer header, exactly as you would to OpenAI.
API_KEY=$(head -1 ~/.config/llama/api-keys)
curl -s http://127.0.0.1:9931/v1/chat/completions -H "Content-Type: application/json" -H "Authorization: Bearer $API_KEY" -d '{"model":"ggml-org/gemma-3-1b-it-GGUF","messages":[{"role":"user","content":"Name three Linux init systems, one line."}],"max_tokens":60}' | python3 -m json.tool
The reply is the standard OpenAI shape. llama.cpp adds a timings block with per-request speed, and system_fingerprint names the build that answered.
{
"choices": [
{
"finish_reason": "stop",
"index": 0,
"message": {
"role": "assistant",
"content": "systemd, OpenRC, SysVinit"
}
}
],
"created": 1794379905,
"model": "ggml-org/gemma-3-1b-it-GGUF",
"system_fingerprint": "b12110-cd959bae9",
"object": "chat.completion",
"usage": {
"completion_tokens": 10,
"prompt_tokens": 18,
"total_tokens": 28,
"prompt_tokens_details": {
"cached_tokens": 0
}
},
"id": "chatcmpl-yq6VKITIpkQSwkq6FdjePHHLq8fbeWSJ",
"timings": {
"cache_n": 0,
"prompt_n": 18,
"prompt_ms": 599.551,
"prompt_per_token_ms": 33.30838888888889,
"prompt_per_second": 30.02246681266481,
"predicted_n": 10,
"predicted_ms": 696.595,
"predicted_per_token_ms": 77.39944444444444,
"predicted_per_second": 12.919989376897624
}
}
Any OpenAI client works the same way: set the base URL to http://<host>:9931/v1 and pass the key as the API key. That is the whole integration story for scripts, editors and agents that already speak OpenAI.
Use the built-in web UI
Open http://127.0.0.1:9931/ in your browser. When the server is remote, reach it through an SSH tunnel such as ssh -L 9931:127.0.0.1:9931 user@server. With a key set, the page opens on “Server Connection Error, Access denied” and an Enter API Key button. Click it, paste the key into the field (1), then press Save & Retry (2):

- 1Paste the key
- 2Save & Retry
Once the key is accepted, the chat view loads. It shows token counts and speed under every message, which makes it a quick sanity check. You should see roughly what llama-cli reported; here it was 13.71 tokens a second.

Stop the foreground server with Ctrl+C before moving on, since the service in the next step binds the same port.
Run llama-server as a systemd service
A server you want to keep running needs three things that a foreground test doesn’t:
- a system user with no login shell
- the model file at a fixed path
- a key file that user can read but not change
Point the unit at a downloaded file with -m rather than -hf, because -hf would cache the model in the service user’s home and download it again whenever that cache is cleared. You also want to know exactly which file a production service loads.
sudo useradd --system --home-dir /var/lib/llama --create-home --shell /usr/sbin/nologin llama
sudo mkdir -p /var/lib/llama/models /etc/llama
sudo curl -L -o /var/lib/llama/models/gemma-3-1b-it-Q4_K_M.gguf https://huggingface.co/ggml-org/gemma-3-1b-it-GGUF/resolve/main/gemma-3-1b-it-Q4_K_M.gguf
sudo chown -R llama:llama /var/lib/llama
openssl rand -hex 32 | sudo tee /etc/llama/api-keys >/dev/null
sudo chown root:llama /etc/llama/api-keys && sudo chmod 640 /etc/llama/api-keys
Check the key file before going further. On Rocky Linux 10 the openssl command wasn’t installed, the pipe wrote an empty file, and the service still started.
sudo wc -c /etc/llama/api-keys
A good file is 65 bytes: 64 hex characters and a newline. If it reads 0, install openssl and run the tee line again. Next, create the unit file.
sudo vim /etc/systemd/system/llama-server.service
Add the following content. On a source build installed under /usr/local, change the ExecStart path to /usr/local/bin/llama-server, which is what the Rocky VM used:
[Unit]
Description=llama.cpp server (OpenAI-compatible API)
After=network-online.target
Wants=network-online.target
[Service]
User=llama
Group=llama
ExecStart=/opt/llama.cpp/llama-server \
-m /var/lib/llama/models/gemma-3-1b-it-Q4_K_M.gguf \
-c 8192 \
--host 0.0.0.0 --port 9931 \
--api-key-file /etc/llama/api-keys
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
The unit binds 0.0.0.0 so other machines can reach it, which means the firewall decides who does, so open it before the service starts. Allow SSH first so enabling ufw can’t lock you out (the OpenSSH profile covers port 22; add your own port if sshd listens elsewhere), then allow port 9931 from your LAN only. The range here is 10.0.1.0/24, so swap in your own.
sudo ufw allow OpenSSH
sudo ufw allow from 10.0.1.0/24 to any port 9931 proto tcp
sudo ufw --force enable
sudo firewall-cmd --permanent --add-rich-rule="rule family=ipv4 source address=10.0.1.0/24 port port=9931 protocol=tcp accept"
sudo firewall-cmd --reload
Reload systemd and start the service now and at boot.
sudo systemctl daemon-reload
sudo systemctl enable --now llama-server
systemctl status llama-server --no-pager
The service came up in about a second and used 217 MB of memory for a 769 MB model, because llama.cpp memory-maps the GGUF file by default as the auto: mmap default in --help shows, so most of the model sits in the page cache rather than in the service’s own memory. In the ufw table you should see 9931/tcp allowed only from the LAN range, next to the SSH rule.

On Rocky Linux 10 with SELinux enforcing, the service ran as unconfined_service_t with the binary labelled bin_t under /usr/local/bin, and ausearch -m avc -ts recent returned no denials. No SELinux change was needed for this layout. We checked only the /usr/local/bin layout under enforcing mode; the tarball path under /opt was not tested on Rocky.
security: no API key is set and CORS allows all origins
That warning in the journal means the server is open, even if you passed --api-key-file. An empty key file doesn’t stop the server from starting; it starts with no keys, logs one warning line and answers every request. On the Rocky VM, /v1/models returned 200 to a request with no key at all while the service listened on 0.0.0.0. Generating a real key and restarting brought back the 401.

Make the 401 test part of every deployment, because it is the only check that proves the key is loaded. Run curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:9931/v1/models and it must print 401, never 200.
Quantize a model with llama-quantize
You only need llama-quantize when the quant you want isn’t published, or after converting a model yourself. Start from the full-precision F16 file, because quantizing an already quantized file is refused (more on that below). Download the F16 GGUF of the same model.
mkdir -p ~/models && cd ~/models
curl -sL -o gemma-3-1b-it-f16.gguf https://huggingface.co/ggml-org/gemma-3-1b-it-GGUF/resolve/main/gemma-3-1b-it-f16.gguf
Quantize it to Q4_K_M, the usual default, and to Q8_0 for comparison. The syntax is input file, output file, then type.
llama-quantize gemma-3-1b-it-f16.gguf gemma-3-1b-it-Q4_K_M.gguf Q4_K_M
llama-quantize gemma-3-1b-it-f16.gguf gemma-3-1b-it-Q8_0.gguf Q8_0
ls -lh *.gguf
Q4_K_M took 21.6 seconds and came out at 769 MB from a 1.9 GB source, the same size as the file ggml-org publishes. The warning that 130 of 340 tensors needed fallback quantization is normal for this model: some tensor shapes don’t fit the K-quant block size, so llama.cpp uses a compatible type for them.

Pick the type from llama-quantize --help, which lists each one with its size and perplexity cost measured on Llama 3 8B. These are the lines most people choose from.
15 or Q4_K : alias for Q4_K_M
15 or Q4_K_M : 4.58G, +0.1754 ppl @ Llama-3-8B
17 or Q5_K : alias for Q5_K_M
17 or Q5_K_M : 5.33G, +0.0569 ppl @ Llama-3-8B
18 or Q6_K : 6.14G, +0.0217 ppl @ Llama-3-8B
7 or Q8_0 : 7.96G, +0.0026 ppl @ Llama-3-8B
The perplexity cost falls fast from Q4_K_M to Q6_K while the file grows by about a third. Q4_K_M is the right start for chat on limited memory, and Q6_K or Q8_0 make sense when the model fits comfortably and accuracy matters more than speed.
failed to quantize: requantizing from type q8_0 is disabled
This is what llama-quantize gemma-3-1b-it-Q8_0.gguf gemma-3-1b-it-Q4_0.gguf Q4_0 printed. Quantizing an already quantized file stacks two rounds of rounding error, so the tool refuses unless you add --allow-requantize. Go back to the F16 or BF16 file instead; use the flag only when no full-precision source exists.
Benchmark with llama-bench
llama-bench runs a fixed prompt test (pp512, 512 tokens of prompt processing) and a generation test (tg128, 128 new tokens), repeats them and reports the mean with a standard deviation. Pass several -m flags to compare files in one run, and a comma list to -t to compare thread counts.
llama-bench -m gemma-3-1b-it-f16.gguf -m gemma-3-1b-it-Q8_0.gguf -m gemma-3-1b-it-Q4_K_M.gguf
llama-bench -m gemma-3-1b-it-Q4_K_M.gguf -t 2,4
If you run the systemd service on the same box, stop it first, or the two will fight over your cores. These are the numbers from the 4-vCPU VM.

Generation speed follows file size. F16 managed 7.99 tokens a second, Q8_0 11.98 and Q4_K_M 13.74, because on a CPU each new token means reading the whole model from memory once, so a smaller file is a faster model. The bigger surprise is the thread count. llama.cpp picked 2 threads on this 4-vCPU VM, which matches its 2 physical cores, yet forcing -t 4 raised Q4_K_M generation from 13.62 to 17.17 tokens a second (26% more) and prompt processing from 42.48 to 49.54. On cloud VMs with hyperthreads, benchmark -t once and add the winner to your unit file.
Offload to an NVIDIA GPU
We did not rerun the GPU path for this update; the numbers in this section come from our July 2026 run on a rented RTX 4090 with an earlier build (b10077) and Qwen2.5-3B at Q4_K_M. The flags below were checked against the current release instead. A GPU build needs the NVIDIA driver and the CUDA toolkit (our NVIDIA driver and CUDA install guide covers both on Ubuntu). The release now ships CUDA 12.8 and CUDA 13.4 tarballs next to the CPU one, and the source route adds one CMake flag, per the project’s build documentation.
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j"$(nproc)"
Older guides add -ngl 99 to push every layer onto the GPU. The current --help lists -ngl as “an exact number, ‘auto’, or ‘all’ (default: auto)”, and --fit is on by default, so the server sizes the offload to the VRAM it finds. Set -ngl yourself only to hold layers back for another process on the card. In July, nvidia-smi showed the 3B model holding about 3.5 GB of the 4090’s 24 GB, and this was the benchmark command, run from a CUDA source build of that era:
./build/bin/llama-bench -m qwen2.5-3b-instruct-q4_k_m.gguf -ngl 99
On the RTX 4090 that run reported this, with the whole model on the GPU:
| model | size | params | backend | ngl | test | t/s |
| ---------------------- | ------: | ------: | ------- | --: | ----: | ---------------: |
| qwen2 3B Q4_K - Medium | 1.95 GiB| 3.40 B | CUDA | 99 | pp512 | 22279.13 ± 3137 |
| qwen2 3B Q4_K - Medium | 1.95 GiB| 3.40 B | CUDA | 99 | tg128 | 327.69 ± 0.77 |
Put next to the CPU run of the same file on an older desktop, the gap is large, and largest where people expect it least.
| Backend (Qwen2.5-3B Q4_K_M, July 2026) | Prompt processing (pp512) | Generation (tg128) |
|---|---|---|
| CPU, Intel i7-6700, 4 threads | 43 t/s | 11.8 t/s |
| GPU, NVIDIA RTX 4090 | 22,279 t/s | 327.7 t/s |
Generation was about 28 times faster on the GPU, and prompt processing about 500 times faster. Generation is limited by memory bandwidth, so the GPU wins by a finite margin. Prompt processing is parallel compute, which a GPU does well, so if you feed long documents the first number is the one you feel. For sizing a card, start with how much VRAM a model needs and our list of the best GPUs for local LLMs.
A dedicated GPU serving many users at once is where llama.cpp stops being the best fit. For high-concurrency serving, our vLLM production install batches requests far more aggressively.
If you run these tools daily, the one-page PDF keeps the flags from this guide next to your terminal.
Choose llama.cpp or Ollama
Ollama’s own README lists llama.cpp as its supported backend, so the choice is about how much of the engine you want to manage, not which one runs models better. In our July run on the same RTX 4090 and the same Qwen2.5-3B Q4_K_M, raw llama.cpp generated 294 to 328 tokens a second against Ollama’s 263, a real but small gap.
| llama.cpp | Ollama | |
|---|---|---|
| Install | Tarball or source build | One-line install script |
| Models | GGUF files or -hf repos | A registry with ollama pull |
| Generation, RTX 4090 (July 2026) | 294 to 328 t/s | 263 t/s |
| API | OpenAI-compatible, port 9931, API key file | Native REST API, port 11434 |
| Tuning | Every flag: threads, context, offload, cache | Sensible defaults, fewer knobs |
Pick Ollama when you switch models often and want them to just work. The Ollama install on Ubuntu, the Ollama commands cheat sheet and the Ollama models cheat sheet cover that path. Pick llama.cpp when you want a small server under your own systemd unit, an exact build you can pin, and flags like -t that bought 26% more speed on our VM. Our Ollama vs vLLM vs llama.cpp comparison adds vLLM to the decision. Whichever you run, keep the API behind a key and a firewall rule, and test for the 401 after every restart.