Serve the model locally
Any server that exposes an OpenAI-compatible/v1 endpoint works. Install one backend and
start it with tool calling enabled. Each backend serves on its own default port, so note the
local URL yours prints. You point your harness at that URL.
Each backend uses its own default port, so your endpoint depends on which one you run.
llama.cpp and MLX use
8080, vLLM and zml/llmd use 8000, SGLang uses 30000, LM Studio uses 1234, and Atomic Chat uses 1337.
The examples in this guide use http://localhost:8080/v1. When you configure a harness,
replace the port with your server’s.Model configuration
LFM3-1.7B is a 1.7B-parameter text model post-trained for agent harnesses such as Hermes Agent, OpenClaw, and Pi. It runs fast on consumer hardware and supports native tool calling, which makes it a good fit for agentic workloads. The two settings worth choosing up front are the quantization and the context length. Both trade memory for quality or capacity, so pick them to fit your hardware. Quantization. Because LFM3-1.7B is small, you have room to trade size for quality. For the GGUF quants, which cover llama.cpp and LM Studio, we recommend starting withQ4_K_M and stepping up to Q8_0 or BF16 depending on your available memory.
MLX uses its own quantization. Pick the 4-bit or bf16 build.
vLLM and SGLang run the full-precision weights on GPU, and zml/llmd runs the FP8 weights.
Need more capability? LFM2.5-24B-A2B is a 24B-parameter
reasoning model with only 2B active parameters per token, post-trained inside Hermes Agent,
OpenClaw, Pi, and OpenCode. It handles longer, multi-step agent tasks and agentic coding more
reliably, at the cost of more memory. The backend examples below, except zml/llmd, work with it if you swap the model ID.
Context length. Agents consume context quickly. If you hit truncation or context-overflow errors mid-run, raise the served
context or trim the agent’s history.
LFM3-1.7B supports up to 256K tokens. The examples serve the full window, but if you’re
memory-constrained, serve a smaller window such as 32K tokens, which is usually plenty for a single agent task.
Start a server
Install one backend and start it with tool calling enabled.- llama.cpp
- LM Studio
- Atomic Chat
- MLX
- vLLM
- SGLang
- zml/llmd
Install:LFM3 requires llama.cpp b11960 or newer. For Linux and build-from-source options, see the llama.cpp guide.Run:The
-hf flag auto-downloads the GGUF.8080 with your server’s port):
Connect your agent harness
Every harness connects the same way: install it, point it at your local server, then run. The examples below usehttp://localhost:8080/v1 and model ID LFM3-1.7B. Replace the
port with the one your server prints and the model ID with the one returned by /v1/models. For zml/llmd, use http://localhost:8000/v1 and LFM3-1.7B-FP8. Only the exact commands differ per harness.
- Hermes Agent
- OpenClaw
- Pi
Docs: Custom / self-hosted providers.Install:Configure:Use the interactive wizard:Or set it directly, then enable tool-use enforcement (without it, the model tends to
describe actions instead of calling tools):Run:
[!Note] Ifweb_searchis missing from the model’s available tools, it may be due tosearchorbrowserbeing listed inagent.disabled_toolsets. Remove both entries inhermes config editand restart Hermes.