Repository navigation
Releases
Splash 1.6.1
Engine
- Faster model startup after an update. On M5 Pro, Flash-Next’s first start dropped from over 5 minutes to 49 seconds. The app and CLI now show warm-up progress.
- Faster prefill and decode. On M5 and later, Qwen3.8-27B MLX 4-bit saw about 25% lower TTFT. Bonsai decode is also faster, and streaming experts now overlaps with compute for 8–17% faster decode on tested Flash-Next and Qwen3.6-35B configurations.
- More room for long prompts and concurrent requests. Splash keeps KV in RAM while space is available and can spill it to SSD during prefill. Flash-Next streams its 28.8 GB PLE table by default, leaving more memory for context and cache.
- More reliable serving. Slow disk reads and blocked console output no longer interrupt requests. Disconnected clients release their slots, and agents can reuse tokenizer prefixes across 16 conversations.
Mac app
- Attach PDFs in desktop or web chat. Pin chats, delete several at once, and switch models from the composer.
- Update or delete individual models, choose a local vision projector or draft, and launch coding agents in the drawer.
- Track live requests and warm-up progress on the Server screen. Supported Qwen models now offer a 1M context option.
Upgrade
- App: choose Splash → Check for Updates… or download the latest DMG.
- Homebrew: run
brew update, thenbrew upgrade splashfor the CLI orbrew upgrade --cask splash-appfor the app.
On M5 and later, the new default --activations auto uses INT8 on selected paths, so outputs can differ from 1.6.0. For bf16 comparisons, use --activations bf16 with a fresh --cache-dir.
Thanks to @dmkjr, @zaiyang-zhang, @jma24, @linson007, @jiiim, @janhilgard, @fishy-ops, @mperreir, @rickhuizinga, @didikrm, and @nailarch.
Splash 1.6.0
Splash 1.6.0
It's been longer than usual since 1.5.0. Sorry for the wait.
This release has few kernel changes. Most of the work went into the engine: how it uses memory, schedules requests, and recovers when something goes wrong. In real serving, with several requests, long contexts, and a memory limit, these decide whether Splash stays responsive. We think they matter more there than another few percent of decode speed.
Serving under a memory limit
Splash 1.6.0 fixes long stalls seen in 1.5.0 under a memory limit. On an M5 Max with 36 GB serving Qwen3.8-Flash-Next Q2_0 at --max-memory 18G, the longest pause in a stream dropped from 300–406 s to 0.92 s. On the same load, prefill throughput rose 1.27x, decode throughput rose 1.25x, and TTFT fell 26%. On a 24 GB Mac serving Qwen3.8-27B UD-IQ3_XXS, TTFT fell 41%. These are concurrent serving results; single-request decode stays within 2% of 1.5.0.
- One memory ledger. Every allocation goes through one ledger, with one order for what to free when memory runs short. Cached data the model can rebuild goes first; memory another request is using goes last.
/statusreports it undermemory_governor. - Idle KV no longer slows every command. 1.5.0 made all cached KV resident for every GPU command, so idle cache slowed decode by up to 5.8x near the GPU's memory limit. Each command now makes resident only the KV it reads. (#644)
- Elastic expert store. With
--expert-memory auto(the default), streamed experts grow as memory allows and give memory back when requests or other apps need it.--expert-memory SIZEstill fixes the size. - SSD cache keeps making room. A full cache replaces old copies instead of refusing new ones.
- Fairer scheduling. A request that has to wait for memory no longer holds up the requests behind it. A long prompt gets its turn after three commands have passed it, as in 1.5.0.
- More context, not less. Qwen3.6-35B-A3B no longer serves less context when given a larger memory budget.
- Smaller prefill chunks use a 64-row tile on M5/M6 and smaller M3/M4 GPUs. (#621)
Reliability
- Engine restarts. A failed engine restarts with a backoff. Restarts stop at once on model files that cannot be served, and after six quick failures in a row (1.5.0 stopped after three). A start that waits for memory keeps retrying.
- Quarantine. A request that crashes the engine twice is refused with
400 request_quarantinedinstead of crashing it again. - Sandboxed PDFs. PDFs render in a sandbox with no network access and no environment variables.
- A request whose KV spills to the SSD no longer fails with
capacity_exhaustedwhile memory is about to free up. (#622, fixes #613)
Models
- Split GGUFs at a repository's root install as one model.
- Qwen3.8-Flash-Next GGUFs with BF16 or F16 PLE tables load.
- Flint GGUFs from lmstudio-community, ggml-org, bartowski, mradermacher, and prithivMLmods install.
Splash for Mac
- Drawer: press ⌘J for terminals and live logs from the server and downloads.
- Smoother app: replies stream without redrawing, long chats scroll steadily, and the composer and controls move together.
- Clearer launch options: short names, an explanation beside each option, and only the options the selected model can use.
- Start failures say why. The server the app started also stops if the app crashes.
- Context ring: a ring beside Send shows how much of the context the prompt uses. (#642)
APIs
POST /v1/chat/completions/count_tokenscounts a Chat request's tokens without running it. (#642)/statusbreaks each request's time to first token down by cause. (#620)- A tool call without a function name is returned as text instead of being dropped. (#632)
- Stop sequences are matched as output arrives. Long stop sequences no longer slow generation.
Fixes
- Thinking budgets shrink with the remaining context to leave room for the answer. A budget cutoff now follows the chat template, so follow-up turns reuse more cached KV.
- With
http_proxyset, the launcher and coding agents now reach the local server directly. 1.5.0 sent the server's API key to the proxy. splash pisends the API key to every Pi version.- A Finder
.DS_Storefile no longer breaks local models. - A pipelined request no longer resets the response.
- The app lists only models whose files are still on this Mac.
Upgrade
- App: choose Splash → Check for Updates…, or download the latest DMG.
- CLI:
brew update && brew upgrade splash. - App installed with Homebrew:
brew update && brew upgrade --cask splash-app.
If you use both the CLI and app, keep them on the same release.
/statusis schema 8. Two fields were removed. Requests ended by a stop sequence count ascompleted.- Stop sequences: each is limited to 16,384 characters. Messages
stop_sequencesmust be an array of nonblank strings; a single string now gets a 400 error. - GGUF screening: a GGUF with tensors the model would not use, such as an attention bias, is refused when it loads.
- PDF input needs
/usr/bin/sandbox-exec. Inside another sandbox, PDF requests get503 document_unavailable. - Persistent cache: a GGUF model's cache starts fresh once after the upgrade. The weights are not downloaded again.
- Downloads are refused only when they don't fit on the disk. The SSD cache shrinks or turns off on its own when space runs short.
Full details are in the 1.6.0 release notes.
Thanks to @linson007, @pcvelz, and @coco-alen for contributions, and to @q-p, @sunnybluesea, and @oinani0721 for detailed reports in #613 and #644.
Full changelog: 1.5.0...1.6.0
Splash 1.5.0
Splash 1.5.0
Splash now has a desktop app: download and run models, chat with them, and connect coding agents from one window. This release also adds local model loading, custom tools in the Responses API, faster decode on M5 and M6, and a fix for image requests that could crash the engine.
Splash for Mac
The desktop app includes its own runtime, shares models with the CLI, and supports:
- Model downloads, switching, and launch settings.
- Streaming chat, saved conversations, images, and reasoning controls.
- Claude Code, Codex, OpenCode, Hermes, and Pi in your chosen project folder.
- Menu bar controls and automatic update checks.
Requires Apple M3 or newer and macOS 26.4 or later. Download Splash, open the DMG, and drag Splash into Applications. Or install with Homebrew:
brew install --cask incoai/tap/splash-appThe app uses the default Hugging Face cache and does not inherit HF_HOME, HF_HUB_CACHE, or HF_TOKEN from your shell. Quitting it stops the model. GGUF files added locally run text only; use the CLI for a separate vision projector or local draft. MLX folders retain their vision support. (#610)
Speed and memory
- Faster decode on M5 and M6. On a 32-core M5 Max, Qwen3.8-27B MLX 4-bit takes 11.7% less time per decode cycle with one request and 7.5–8.3% less with two to four. The 27B UD-Q4_K_M takes 2.2–4.4% less. On a 12-core M6, the 35B UD-Q2_K_XL takes 0.4–1.8% less. These are decode-cycle measurements; per-token gains for the batched 27B MLX are 1–2 percentage points smaller at three and four requests. Other GPU core counts were not measured end to end. (#235)
- Lower MoE router memory. The 35B MLX 4-bit keeps its router quantized, saving 62 MB and reducing router reads from 84 MB to 22 MB per decode cycle. Quantized GGUF MoE routers also load; F16 routers remain unsupported. (#599)
- Faster quantized kernels on M3, M4, and M5, including MLX 4-bit and several GGUF formats. These are kernel-level improvements; whole-model gains vary. (#600, #602)
Models and APIs
-
Local model files: serve a GGUF file or MLX directory with
--model-dir, without contacting the Hub. Use--draft-modelfor a local DFlash2 draft and--mmprojfor a GGUF vision projector. Otherwise, the matching draft must already be cached. (#338)splash serve --model local/qwen \ --model-dir /models/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-Q4_K_M.gguf \ --draft-model /models/Qwen3.8-27B-DFlash2
-
Custom tools: Responses accepts freeform tools such as Codex's
apply_patchand returnscustom_tool_callitems. Tool grammars are described to the model, not enforced; streamed tool input arrives when the call closes. (#426) -
Request timings: Responses and Messages report prompt/decode counts, latency, rates, cached tokens, and draft counts. Messages also reports thinking-token usage. (#427)
-
Reasoning discovery:
/v1/modelsreports supported reasoning efforts and defaults. (#610) -
Pi prefix caching:
splash pienables later system messages when the model's template supports them, preserving reusable prefixes. (#609) -
Shell completion: expands to 11 suggested models and completes local model, draft, and projector paths. (#338, #610)
Fixes
- Image requests no longer trigger GPU timeouts on models whose experts stream from SSD. Image encoding runs before prefill in a separate GPU command, and image requests use Neural Engine prefill again. (#601, fixes #596)
- Ctrl+C stops model downloads promptly. The next download from that repository cleans up interrupted partial files. (#610)
- Browser chat asks for an API key only after a 401 response. (#610)
- Update filelock to 4.2.4. (#603)
Upgrade
brew upgrade splashKeep the CLI and desktop app on the same release.
- Use full CLI option names: abbreviations such as
--max-memare no longer accepted. - Cached models: GGUF metadata is rebuilt from existing weights without downloading them again. Persistent caches for GGUF and MLX MoE models start fresh once; dense MLX caches remain usable. Switching between different App and CLI releases can repeat this.
- Offline startup: if an older Splash downloaded your model and draft, run
splash serve1.5.0 for that family once with the Hub reachable before using it offline in the app or through--model-dir. - API clients: Responses and Messages add
timings; Messages addsoutput_tokens_details.thinking_tokens. Responses'input_tokens_details.cache_write_tokensis always 0; compute uncached input asinput_tokens - cached_tokens. - Numerical changes: kernel summation changes can slightly alter greedy output. On M5 and M6, output can also depend on concurrent requests.
Thanks to @zaiyang-zhang, @yannip1234, @ghisguth, and @lucaslie for contributions, and @shimbara for the detailed report. The zero-point kernel work builds on @ghisguth's #569.
Full changelog: 1.4.1...1.5.0
Splash 1.4.1
Splash 1.4.1
Splash 1.4.1 adds Ornith-1.5-122B and Flint, improves MoE performance on base M3/M4 Macs, and fixes startup and request stalls under memory pressure.
New models
Ornith-1.5-122B loads from GGUF or MLX with its matching DFlash2 draft selected automatically (#539, #560).
splash serve --model unsloth/Ornith-1.5-122B-GGUF:UD-Q4_K_XLGGUF supports vision. ornith-ai/Ornith-1.5-122B-MLX-4bit also includes its vision tower; --language-only is optional.
Flint, RecursiveSI's 40B-A2.5B MoE, serves from GGUF with automatic incoai/Flint-DFlash2 selection, tool calling, structured output, and up to 262,144 tokens of context (#573–#576).
splash serve --model unsloth/flint-GGUF:UD-Q4_K_XLFlint is text-only and decides when to reason itself; reasoning settings cannot force it on or off. MLX checkpoints are not supported.
Pruned-expert GGUFs now load, including ISTA's Flash-Next GSQ-RCO-Coder variants (#561).
Speed and memory
- Flint measured 190 tok/s across ten mixed prompts on an M5 Max with 36 GB, versus 89 tok/s for llama.cpp on the same Mac and UD-Q4_K_XL checkpoint. These are the pooled results from four runs (#576).
- Flint releases sliding-window K/V once it is no longer needed. With INT8 KV, cached pages beyond the window use 71% less memory; full-history attention is retained (#574).
- Faster MoE kernels on base M3/M4 GPUs with ten cores or fewer. On the ten-core M4, most tested formats took 12–45% less GPU time per MoE layer in small-batch benchmarks. This is a kernel measurement, not an end-to-end speedup (#578).
Fixes
- Low-memory startup: release idle warmup buffers before checking the host memory reserve, avoiding false startup refusals on 24 GB Macs. Startup failures also report their actual reason instead of a generic EOF error (#577, #563).
- Large images: fix GPU timeouts when the 27B's image encoding overlaps ANE prefill. Image-encoding commands run on the GPU; subsequent text chunks still use ANE (#593).
- Concurrent requests: fix queued requests stalling under memory pressure while another request grows its KV cache, while preserving capacity for long single requests (#579).
- Accept split GGUFs whose first file contains only metadata (#560).
- Report missing MLX vision shards clearly and suggest
--language-onlyfor affected checkpoints (#568). - Update Pillow to 13.0.0 and certifi to 2026.10.19 (#558).
- Improve model verification checks, including penalty and warmup-EOS checks, memory handling, and stale weight records (#562, #564–#567).
Upgrade
Existing GGUF installations refresh derived metadata locally on first start, reusing downloaded weights. Their persistent conversation cache starts fresh because the model identity changes. Existing MLX cache layouts remain compatible when model and draft identities are unchanged.
Thanks to @zhijian-liu, @ezoushen, @wmTJc9IK0Q, @linson007, @cresslank, @Litee and @realzop for contributions, reports and testing.
Full changelog: 1.4.0...1.4.1
Splash 1.4.0
Splash 1.4.0 adds Qwen3.8-Flash-Next, SSD streaming for experts and PLE, active KV spill, and up to 1M context. Adaptive draft verification and mixed structured-output batching improve decode throughput.
Qwen3.8-Flash-Next
Flash-Next GGUFs now load with QSA sparse attention and an automatically selected DFlash2 draft (#467–#469):
splash serve --model ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF:Q2_0Unsloth's Flash-Next GGUFs also load. Flash-Next MLX checkpoints are not supported yet.
On an M5 Pro (20-core GPU, 64 GB), Q2_0 measured 95 tok/s decode, 1,312 tok/s prefill at 32K, 171 ms cached TTFT, and 171 tok/s aggregate decode with four concurrent requests. These use selected SPEED-Bench coding prompts over HTTP with 1,024-token outputs; prefill rate uses time to the first token, and all 16 replayed prompts hit the cache (#521).
SSD streaming
The planner shares one memory budget between weights, context and streaming caches (#463, #468, #474–#481, #495):
- MoE experts: GGUF and MLX targets keep experts resident when they fit, otherwise stream them from the SSD.
--expert-memory SIZEsets their RAM store;offloads every expert. - Flash-Next PLE: the n-gram table streams when it cannot fit, with a default 512 MiB row cache.
--ple-memory SIZEchanges that cache;offloads the whole table. - Active KV: Flash-Next can spill a running request's KV to the SSD, keeping its sparse index and hot blocks in RAM.
--sparse-kv-memory SIZEcontrols the hot cache. The 27B and 35B keep active KV resident.
Streaming is automatic and trades SSD reads for RAM. KV spill requires a writable SSD cache with available quota. --max-memory leaves room for other applications and shrinks the automatic stores. Streamed prefill runs in passes so other requests can decode and cancellations take effect between passes.
Up to 1M context
All three families support static YaRN beyond their native 262,144-token window. Add --max-context 1M to enable it; 1M means 1,000,000 tokens, with an upper limit of 1,010,000 (#431, #469, #518).
The requested context must fit available RAM and SSD resources. YaRN stays off within the native window and changes outputs even on short prompts when enabled, so request the longer window when needed.
Faster decode
MoE targets adapt how many draft tokens they verify, avoiding work on unlikely proposals (#487). Against verifying every row, Flash-Next Q2_0 with a 12G expert store gained 13–38% decode throughput at one to four concurrent requests on an M5 Max 36 GB. The 35B UD-Q2_K_XL with a 4G store gained 6.5–42% on an M6 24 GB.
JSON Schema and ordinary requests now share decode batches (#520). With two of each on an M5 Max 36 GB, aggregate throughput rose 349 → 457 tok/s for the 35B MLX 4-bit and 138 → 210 tok/s for the 27B MLX 4-bit. Single-request throughput was unchanged within noise.
These comparisons measure individual changes. Metal optimizations also improve MoE projections, gathering, low-bit decode and sparse attention.
Other changes
- Preserve conversation replay states during routine memory reclamation, and durable restart points when the SSD cache fills. Fix spill-resume accounting, pinned-cache lifetime errors and engine relaunch with spilled KV (#484, #512, #513, #524).
- A single unexpected exit no longer clears persistent cache; repeated failures or engine-reported execution faults during startup probation still can (#498).
- Honor explicit
--max-cache-disk SIZEquotas subject to the volume's free-space floor. Fix a host-step race that could cause GPU timeouts, and avoid watchdog restarts while streamed commands make progress (#479, #494, #515). - Load split GGUFs, Q2_0 and additional GGUF norm/router/gate tensor types. Add network-free
install.local.prepare_local_model()for local checkpoints (#451, #452, #500). - Normalize tool-schema type aliases and preserve untyped arguments as text. Bound schema/prompt caches by bytes; PDF preparation no longer holds up unrelated requests while waiting (#514).
- Remove stray
</think>tags after reasoning ends; report streaming, KV spill and verification mode in/status(#486, #481, #487).
Upgrade notes
- Default
--verify-rows autocan change MoE greedy and seeded outputs as timings and server history change. Use--verify-rows allfor repeatability with isolated requests and identical cached prefixes. - Affected weight layouts, including the 27B and 35B MLX checkpoints, start their persistent cache cold once; it refills during use.
- Direct
python -m server.serverintegrations should drop--tokenizerand useprepare_local_model()(#500).
Thanks to @janhilgard and @Vardaan97.
Thanks also to Strata for inspiring some of the design choices in this release.
Full changelog: 1.3.2...1.4.0
Splash 1.3.2
Splash 1.3.2 adds NVFP4 and FP8 checkpoints, more GGUF layouts, optional BF16 recurrent states and thinking budgets. The SSD prefix cache is now on by default, with faster prefill and lower TTFT.
Models
These changes apply to Qwen3.8-27B and Qwen3.6-35B-A3B checkpoints and compatible fine-tunes:
- Load MLX
nvfp4, NVIDIA Model Optimizer and compressed-tensors NVFP4/FP8 checkpoints, and llama.cpp NVFP4 GGUFs. For example:nvidia/Qwen3.8-27B-NVFP4andmlx-community/Qwen3.6-35B-A3B-nvfp4(#356). - Load more GGUF variants, including
UD-Q8_K_XL, Q5_0/Q5_1 weights, IQ1_M and IQ2_S token tables, other previously refused embedding types, fused MoE gate/up experts and mixed-type GDN alpha/beta tensors. Quantized checkpoints can now contain BF16 and F16 weights; whole BF16, F16 and F32 targets remain unsupported (#379, #403; fixes #375 and #394).
Memory and caching
--state-format bf16halves the GDN recurrent-state storage. A cached state shrinks from about 153 to 81 MiB on the 27B and from 64 to 34 MiB on the 35B. Computation stays FP32, with rounding when BF16 states are stored; FP32 remains the default (#385).- The SSD prefix cache defaults to
auto, with a quota up to 32 GiB that follows available space on its local volume.--max-cache-disk 0turns it off. The cache adjusts as the volume fills and can keep serving after a full-disk write; unused persistent caches are cleaned up after 14 days (#408). --persistent-cachekeeps the SSD cache across restarts and remains off by default. It now works without an explicit disk quota.--max-cache-disk-writes SIZEreduces optional writes to a daily budget; conversation resume states, their KV pages and the clean-stop flush can exceed it. Writes have no budget unless one is set (#408).- The RAM prefix cache grows only into spare memory and gives memory back when macOS warns. Idle reclaim avoids repeatedly discarding an extra recovery margin when no request needs it (#383, #418; fixes #325).
POST /v1/memory/releasefrees idle model weights on demand while keeping the prefix cache. It works with--idle-release off; the next request restores the weights. A busy engine or a restore in progress returns 409engine_busy, and 200 confirms the weights are released (#374, #416).cache_saltisolates KV and cached states between different salts, including on disk. Chat/text completion metrics and the Done log now report KV reused beyond a request's initial cached prefix asdeduplicated_tokens, including reuse after suspension (#382, #415; fixes #368).
Thinking and agent compatibility
- Limit reasoning with
thinking_token_budgetin Chat Completions and Responses, or set a server default with--thinking-token-budget N. The default is uncapped;-1disables the cap for a request. Messages honors enabled thinking's requiredbudget_tokens(at least 1024) and accepts budgets at or abovemax_tokens(#397, #411; fixes #373). --thinking-end-text TEXTcan turn the model toward its answer when the budget runs out. The text must end with</think>and fit within 64 tokens. With structured output or constrained tools, the cutoff waits for a complete character before closing thinking, avoiding grammar failures on split Unicode characters; reasoning can therefore run a few tokens past the budget (#405, #421).- Messages applies Claude Code's mid-session
tool_additionandtool_removalblocks in conversation order, including additions by reference. Later definitions replace tools of the same name (#391; fixes #389). - Tool names can contain dots, spaces, punctuation and Unicode, up to 256 characters within the call format's rules. Non-strict tools with unusable JSON Schema descriptions no longer reject the whole request; strict tools still require valid schemas (#400; fixes #392).
- Output limits larger than the remaining context now generate up to the window's end instead of returning 400. Optional null fields,
seed: -1, stop strings alongside tools or structured output, and malformed historical tool arguments are handled without rejecting otherwise usable requests. Reasoning history and template options are accepted in more client formats; context-management fields do not make Splash edit or compact the supplied history (#401, #402).
Latency and decoding
Individual-change benchmarks on a 40-core M5 Max measured:
- Sending an unconstrained request's first token as soon as prefill selects it reduced short-prompt median TTFT by about 37 ms on the 27B and 13 ms on the 35B (#358).
- Capturing cached states inside prefill commands reduced prompt-processing time for follow-ups adding 100–300 tokens to a cached 8K prefix by 11–14% on the 27B and 20–23% on the 35B (#406).
- Vision encoding of a synthetic 16,384-patch image fell from 1.419 to 1.141 seconds, about 20%, with the 27B vision tower (#384).
MoE decode skips a redundant gather for one-lane staged plans: measured 35B cycle time fell 1.4–2.5% on M5 Max, M5 Pro and M6. GDN decode folds retained rows into the next step instead of running a separate commit, improving most tested cases by up to about 5.5%; its M3 Max one-lane 35B cycle was 0.4–0.8% longer (#365, #366).
Status and reliability
/status.active_requestslists each live request's phase, priority, prompt progress, generated tokens and age. Started logs show effective reasoning settings; per-request speculative decoding counts appear in Chat/text completiontimingsandmetrics.speculation, with tokens per cycle on the Done line (#255, #357, #395, #399).- Cancelled, expired or failed prefill commands keep their previous checkpoint until a successor is published, so retries retain their recovery point (#420).
- Non-finite prefill results are kept out of the cache, damaged restored states are invalidated, and NaN/negative-infinity logits cannot become selectable through penalties. Failure messages report the invalid logit counts (#381, #396).
- Engine relaunches preserve the first engine's context when the memory plan can hold it, preventing a changed ANE calibration from leaving the server permanently unavailable (#424).
- Truncated emoji and other lone surrogate characters tokenize as U+FFFD, including in scoring prompts. Responses keeps a completed tool call's status intact when a stop string cuts a later call; Messages correctly rejects unsupported assistant prefills even after dropped system messages (#354, #398, #412, #413).
- Request-start logging no longer blocks token delivery on the runtime reader thread. GPU tape-offset overflow is rejected before submission, and startup reports obsolete prepared-weight files from Splash 1.1 that can be removed (#369, #380, #414).
Thanks to @janhilgard, @linson007 and @daijiong1977 for their contributions.
Upgrade
brew update
brew upgrade incoai/tap/splashRestart the server after upgrading: the native/server protocol changes from 8 to 10.
The default SSD cache reserves a state-staging buffer, so memory-limited Macs may advertise a smaller automatic context. Use --max-cache-disk 0 to turn it off. Existing FP32 persistent caches remain reusable when model and cache layouts match; BF16 states use a separate namespace.
Full changelog: 1.3.1...1.3.2
Splash 1.3.1
Splash 1.3.1 moves MLX targets and DFlash drafts onto the same block kernel library as GGUF targets, loads more checkpoints of the two supported model families, and reduces conversation-cache memory.
Breaking change: Splash packages
Splash packages (incoai/Qwen3.8-27B-Splash, incoai/Qwen3.6-35B-A3B-Splash and community packages) no longer load (#335). The MLX models below load the same weights as the two official packages:
splash serve --model mlx-community/Qwen3.8-27B-4bit
splash serve --model mlx-community/Qwen3.6-35B-A3B-4bitFor a fine-tune, use its own MLX or GGUF release. For an installed package, the error names its Hub cache folder, which can be deleted.
Models
These changes apply to Qwen3.8-27B and Qwen3.6-35B-A3B checkpoints and compatible fine-tunes:
- MLX affine formats at 2, 3, 4, 5, 6 and 8 bits, with groups of 32, 64 or 128, now load, including mixed precision. MLX mxfp4 checkpoints also load. For example:
mlx-community/Qwen3.8-27B-8bit(#336). - GGUF repositories with an F16 vision projector can serve images. The projector must preserve the BF16 weights accepted by Splash's vision loader (#330).
- MLX repositories that save the image processor in
processor_config.json, as newer Transformers releases do, now serve images (#353). OptiQ checkpoints install with--language-only(#349).
Speed and memory
All MLX targets and drafts now use the block kernel library used by GGUF targets (#345, #346). In the release check against 1.3.0, on a 40-core M5 Max:
- With 2–4 requests at once, the 27B MLX 4-bit target uses 4–26% less GPU time per decode step. Single-request GPU time is effectively unchanged.
- The tested 27B and 35B
UD-Q4_K_MGGUF targets use about 6–12% less GPU time per decode step at batch sizes 1–4. - The 27B and 35B MLX 4-bit targets use 3–16% less GPU time in the measured 4K-token cold and 544-row prefill runs.
On a 40-core M3 Max, the 35B MLX 4-bit target uses 8–18% less GPU time per decode step. Output tokens and draft acceptance matched 1.3.0 in all 45 backend-regression cases on each Mac.
A cached conversation state takes 153 MiB instead of 187 MiB on the 27B, and 64 MiB instead of 109 MiB on the 35B. The same cache budget can hold more conversation states (#329).
Other changes
/statusand/metricsreport the Mac's thermal state, and the log records each change, making heat-related slowdowns visible (#339)./v1/messagesand/v1/messages/count_tokensaccept Claude Code tool search'stool_referenceblocks inside tool results (#342, fixes #337).- The chat page renders Markdown and highlights code, pauses automatic scrolling while you read earlier output, and displays reply images as links. It also adds reply and code copying, individual conversation deletion, a collapsible sidebar and a bounded Thinking area (#343).
- After an idle release, restoring weights waits for memory instead of stopping the engine on an allocation refusal. A restore that remains blocked for 30 seconds returns a retryable 503; the engine stays available for the next request (#347).
Thanks to @komarov-dc, @linson007 and @shixianqin for their contributions.
Upgrade
brew update
brew upgrade incoai/tap/splashRestart the running server after upgrading. If it used a Splash package, change --model to an MLX or GGUF checkpoint as described above.
The persistent conversation cache written by 1.3.0 is not reused because cached states changed format. It refills as you use Splash; existing MLX and GGUF weight downloads can be reused.
Full changelog: 1.3.0...1.3.1
Splash 1.3.0
Splash 1.3.0 accelerates Qwen3.8-27B prefill by running its feed-forward layers across the GPU and Apple Neural Engine, reducing time to first token for long prompts. It also improves shared-prefix reuse, burst serving, model validation and agent connections.
Neural Engine prefill
- Enable GPU/Neural Engine FFN splitting by default for supported dense targets. The engine calibrates the split for each Mac and model, selects the prompt sizes that benefit, and remembers the result. Short prompts, Qwen3.6-35B-A3B MoE and decoding remain on the GPU.
- Use Hadamard-rotated W8A8 computation for the Neural Engine's part, staging weights from the existing model rather than keeping an int8 copy of the whole model.
- Stop the split and rerun the affected chunk on the GPU after an evaluation failure, timeout or non-finite result. An automatically selected split also stops if it loses to the GPU alone.
splash serve --disable-aneselects the GPU-only path. - Report the split's state, share, minimum prompt-chunk size, evaluations, timing and reruns under
ane_ffnin/status. Unload the Neural Engine program at idle release and reload it for the next request.
Measured cold-prompt TTFT for Qwen3.8-27B, one request:
| Mac | Weights | Input tokens | GPU only | GPU + Neural Engine | Speedup |
|---|---|---|---|---|---|
| M5 Max | MLX 4-bit | 14,096 | 14.09 s | 12.36 s | 1.14x |
| M5 Max | UD-Q4_K_M GGUF | 14,096 | 13.74 s | 12.24 s | 1.12x |
| M5 Pro | MLX 4-bit | 14,096 | 28.28 s | 21.57 s | 1.31x |
| M6, 24 GB | UD-IQ3_XXS GGUF | 14,096 | 45.77 s | 30.8–31.3 s | 1.46–1.49x |
| M3 Max, Low Power Mode | MLX 4-bit | 14,096 | 91.04 s | 62.43 s | 1.46x |
| M4 Max, 40-core GPU, 128 GB (community) | Q8_0 GGUF | 14,328 | 77.6–78.1 s | 40.7–45.9 s | 1.77x median |
The M4 Max community benchmark tested the final PR revision 80827e1, whose production engine sources match this release. It also measured 1.71x at 2,040 tokens, with TTFT falling from 8.5–9.2 s to 5.2–5.4 s. That run measured prefill, not decode or accuracy. The other rows are measurements recorded in the merged implementation's tests.
These are prefill measurements, not decode speedups. First startup can take longer while programs compile and the split calibrates. The split consumes memory that would otherwise hold KV cache: the measured 24 GB M6 automatic context was 61,433 tokens versus 73,721 with --disable-ane. An explicit context limit can reduce the split or select the GPU alone. The W8A8 path is numerically different from GPU-only execution; startup checks each program function against the GPU. The private ANE interface falls back to GPU-only execution when unavailable.
Shared-prefix caching
- Keep the last prefill checkpoint beside a completed prompt's replay state when space permits, so the next conversation sharing a long prefix can resume before its point of divergence instead of prefilling from zero (#301).
- In a measured 19K-token shared-prefix case, the second conversation's TTFT fell from 39.6 s to 6.3 s, reusing 16,384 tokens. This retains one additional reclaimable state per sufficiently long prompt (187 MiB on the 27B); with SSD caching, eviction can write that state to disk.
Serving and APIs
- Admit incoming connections without creating a thread for each connection; use a fixed worker pool and deliver retryable 503 responses under overload instead of resetting burst connections.
- Return errors in the requested API's format, including early Anthropic Messages errors. Return server faults as server errors rather than misclassifying them as invalid JSON; include
Retry-Afterfor retryable systemone errors. - Bound JSON depth to 128 levels and tool/response-format schema depth to 64. Improve schema caches, cached PDF access, disconnect handling and answer-slot preparation for large systemone prompts.
- Keep
/statusreads from changing memory admission hysteresis, and report held requests using admission's own decisions.
Models, agents and compatibility
- Check model configuration and quantization using the engine's own rules before downloading weight files. Improve errors for unsupported models and malformed GGUF/configuration data, and validate the exact buffer and weight extents GPU kernels read.
- Add
--portto all five agent launchers, fixing connections to servers on non-default ports (#311). Put an agent's own--portafter--. - Keep API key values out of generated Hermes profiles, serialize Pi configuration updates, and let another model's download proceed without holding the environment setup lock.
- Publish Hub snapshot pins by atomic rename, fixing failures on filesystems such as ExFAT that do not support hard links.
- Tune selected single-lane paired N256 projection plans; improve persistent-cache restart coverage, installed-client checks and native build coverage in CI.
Upgrade
The native/server protocol changes from 7 to 8. Restart existing Splash servers after upgrading so both components run the new version. Existing installed GGUF models refresh their derived metadata at the next start without downloading their weights again.
Full changelog: 1.2.1...1.3.0
Splash 1.2.1
Splash 1.2.1 fixes requests failing across Mac sleep, preserves tool-call arguments, and makes idle memory release configurable.
Tool calls
- Fix tool fields being renamed or dropped when the model writes parameters in a different order, and string/number unions losing values such as
12:00or0800(#293, #294). - With automatic tool choice and no strict tools or response format, let the model write calls without an output grammar. Required/named tool choice, disabled parallel calls, strict tools and structured response formats retain their applicable constraints.
- Read arguments according to the chat template and the tool's declared JSON types. Preserve parameter contents containing tool-format tags, multiple calls and text after calls; a tool-call opening also ends unclosed reasoning.
- Return generated arguments to the client without post-generation schema validation; the client handles tool argument errors. Request schemas are still checked before generation, and strict tools retain schema constraints. Repeated parameters keep their first value so streaming and complete responses agree.
- Allow
ignore_eoswith unconstrained tools; it remains incompatible with output grammars.
Mac sleep
- Measure engine timeouts, keep-alives and execution durations using awake time. A sleep no longer consumes the Metal command watchdog, resource waits or idle memory timers (#275).
- Prevent automatic system sleep while requests run. The display may still sleep;
--allow-idle-sleepopts out. Explicit sleep and lid closure are still allowed. - A real-model request resumed after a four-minute sleep without an engine restart. Deep standby lasting hours and driver-reported GPU failures remain outside this fix. An explicitly timed request still in transit to the engine when sleep begins can expire on arrival.
Memory and timeout options
- Add
splash serve --idle-release DURATION|off(#292). The default remains10m; examples include30m,2hand90s.offkeeps weights allocated and backend buffers wired between requests. KV memory reclamation under pressure remains active. - Expose
weights.idle_release_seconds,weights.releasedandweights.restoresin/status; the interval isnullwhen idle release is disabled. - Accept seconds or an
s,morhsuffix for--request-timeout, using the same duration format as idle release. Existing numeric values remain seconds; omitting the option still means no request timeout.
Thinking and chat templates
- Pass
reasoning_effort: "none"as well asenable_thinking: falsewhen reasoning is disabled. This fixes non-thinking Chat and Anthropic Messages requests on Nex-N2.5-mini, an existing Qwen3.6-35B-A3B-compatible fine-tune; judgments and systemone use the same template options. - Read the request's
chat_template_kwargs.enable_thinkingonce: a boolean overrides the effort,nullfollows the request/server effort, and other values return 400. Explicittruewith effortnoneenables thinking at the template's default effort.
Full changelog: 1.2.0...1.2.1
Splash 1.2.0
Splash 1.2.0 adds persistent conversation caching, OpenAI text completions, more sampling controls, and broader GGUF compatibility. It also unifies Metal memory residency, replaces sparse KV mapping, improves concurrent serving and memory handling on 24 GB Macs, and removes the extra prepared-weight copy on disk.
Persistent conversation cache
- Add
--persistent-cachewith--max-cache-diskto reuse durable conversation prefixes after a clean restart, a compatible upgrade, or a crash.--cache-dirselects the location; the default is~/Library/Caches/Splash/prefix-cache. - Persist recent conversation restore points in the background, including points still in RAM. A clean shutdown flushes pending restore points for up to six seconds. Crash recovery restores points that reached disk, rather than guaranteeing every recent token was saved.
- Isolate caches by model manifests and cache layouts, lock each namespace, bound storage by the configured quota, and validate recovered data. Invalid or incompatible entries fall back to recomputation.
- Persistence is off by default.
--max-cache-diskalone keeps its temporary, session-local behavior. Persistent mode can write more to SSD when RAM is plentiful.
Metal memory residency and KV cache
- Extend the residency management previously used for weights to every backend buffer: weights, KV extents, recurrent state, draft rings and scratch. Buffers join one residency set attached to the command queue and stay wired between requests during active serving, avoiding repeated residency declarations per dispatch. This does not keep weights resident indefinitely: residency lapses after ten minutes without a command, and idle weight memory is explicitly released as described below.
- Replace placement-sparse KV buffers and 64 KiB heap map/unmap operations with ordinary shared Metal buffer extents, reached through GPU-address page tables. Remove the sparse mapping queue, event waits, mapping timeout and startup sparse-support probe, eliminating that uninterruptible mapping path during serving and teardown.
- Move SSD KV eviction and restoration directly between shared extents and the disk tier on its I/O worker. Remove the GPU staging ring, KV-copy kernel and copy-only GPU commands; SSD-offload configurations regain the staging ring's memory, accounting for the state staging buffer.
- Compact live KV pages into other extents so empty extents can actually be released to macOS. Memory-pressure recovery now returns allocated memory, rather than only evicting cached pages inside still-allocated extents.
- Stop submitting GPU work during backend teardown. Residency expiry uses a precompiled minimal compute kernel instead of a lazily compiled blit, improving shutdown and recovery behavior.
APIs, sampling and structured output
- Add
POST /v1/completionsfor a raw text prompt or token IDs, with streaming and usage reporting. This initial endpoint supports one completion; prompt batches, suffix, echo, logprobs, best-of and multiple choices are unsupported. Its default output limit is 16 tokens. - Add
ignore_eosto Chat and text completions. It suppresses model stop tokens while respecting explicit stop strings and the output budget; it cannot be combined with tools or structured-output grammars. - Add
presence_penalty,frequency_penalty,repetition_penaltyandmin_pto Chat, text completions and Responses. Replace the previous top-32 candidate sampling path with exact top-k/top-p over the whole vocabulary, including under DFlash speculative decoding. Arbitrary top-k values are accepted;0and-1disable the top-k filter. Omitted penalties remain neutral. Non-emptylogit_biasremains unsupported; null and an empty object are accepted. - Enforce required and named
tool_choiceand strict tool schemas across Chat, Responses and Messages. Fix schema-pattern propagation, bound constrained whitespace and referenced patterns, and reject unsupported schemas before prefill. - Accept
chat_template_kwargs, includingenable_thinking, with checks for reserved fields and conflicting reasoning settings. The built-in chat page now defaults to the server's configured reasoning effort. - Preserve text emitted after tool calls and the chronological block order in Messages and Responses. Fix incremental tool-argument streaming, whitespace around tool-call framing, partial calls at the output limit, and streamed/non-streamed output consistency. Remove leading answer newlines immediately after
</think>and avoid replacement characters when a token limit cuts a UTF-8 character.
Serving and client behavior
- Add
--decode-shareto balance active decoding against contended prefill. The default is0.5;0retains one-command alternation. Improve short-prefill completion, priority-based lane preemption and resumption, and scheduling while other lanes wait for disk I/O or grammar masks. - Add repeatable
--allowed-originfor explicitly permitted browser/app origins, including custom schemes used by desktop apps. - Add
--announce-served-nameso API responses report the first--served-model-namealias./v1/modelslists it first;/statuscontinues to identify the loaded model. - Add
splash serve --offlineto start an already installed model without the Hub probe. It does not download missing models. - Forward
--request-timeoutand--queue-sizethroughsplash serve. Requests no longer have a default 1,800-second timeout; an explicit timeout remains available. - Chat and Responses requests that omit an output limit can use the remaining context instead of an implicit 32,768-token cap. Explicit limits receive clearer context-capacity errors. Messages reports
model_context_window_exceededwhen context is exhausted; text completions retains its 16-token default. - Add Fish completion for commands and installed models, installed by Homebrew alongside Bash and Zsh completions.
- Run Hermes in a dedicated profile under the user's Hermes root and fix argument forwarding. Configure clients from
/v1/modelswithout waiting for/status. Pi receives a 32K output allowance, lowered by Pi to the remaining context; OpenCode and Hermes retain a quarter-window allowance for contexts below 128K. - Give the built-in chat page a favicon and keep it from scrolling or changing its waiting state on empty keepalive chunks.
Model loading, memory and performance
- Load MLX, GGUF, draft and vision weights directly into Metal buffers from the original model files at each start. Remove the extra prepared-weight disk copy and its preparation cache. After ten idle minutes, release weight memory and reload from the original files on the next request, while continuing to answer status and cancellation messages.
- Support GGUF token/embedding tables stored as
IQ4_XS,IQ4_NLorIQ3_S, and quantized GDN alpha/beta gate pairs supported by the loader. Accept the legacy RoPE type key and whole-number floating values in model configurations. - Probe Apple GPU family 11 explicitly. Improve affine Q4 split-K execution on Apple10/Apple11 (M5/M6), reduce decode CPU/driver overhead, prepare sampling pipelines at startup, and improve shared output-head and draft execution.
- Fix partial-tile GGUF prefill synchronization and Q4 prefill shader validation on macOS 26.5. Fix vision FP32 residual parity.
- Improve serving on memory-squeezed 24 GB Macs: preserve a minimal serving footprint during memory warnings, return empty KV allocations to macOS, allow admitted requests to grow within the configured budget, and avoid repeatedly restarting queue wait limits.
- Keep the reusable conversation state before the chat template's generation prompt, and prioritize unfinished requests' replay points for the next turn. Improve checkpoint/branch-point placement and prefix leases, and let shared-prefix requests wait for a producer restoring from SSD. Close admission behind requests waiting for memory so later small requests cannot indefinitely bypass them.
- Allocate vision scratch dynamically for the images being encoded instead of reserving its maximum working set at startup. Share image embeddings across requests and retained states, skip images already covered by a restored prefix, and reclaim unused vision arenas before useful embeddings. The advertised text context no longer pays the former fixed vision-scratch reserve.
- Fix SSD slot reclamation and quota accounting, including shrinking files and punching freed slots. Preserve a state's only copy when a disk restore cannot be promoted into the RAM cache. Disk-full or unwritable-tier failures disable affected writes while serving continues; filesystem allocation can briefly exceed the logical quota until freed slots are punched.
Reliability and compatibility
- Recover engines that fail or stop responding even when idle. Bound repeated crash/restart loops, report retryable failures while recovering, and return
engine_failedif recovery stops./readynow follows engine readiness without treating an ordinary busy tick as a failure. - Isolate request-level sampling failures, keep native input and cancellation responsive, bound grammar work, cancel image/document preparation on deadlines, and discard status from replaced engines. A grammar mask left unanswered for five seconds fails that request with retryable
mask_timeoutinstead of holding its batch./readyremains healthy under memory warning and returns 503 under critical pressure. - Return JSON HTTP errors for malformed requests, accept matching Bearer and
x-api-keycredentials together, and improve responses to overloads, stalled uploads and connection bursts. Full queues return 503 withRetry-After. - Messages requests ending in an assistant prefill now return 400 rather than silently starting a new assistant turn.
- Seeded sampled output can differ from earlier versions because draft proposals and sampling execution changed; a seed does not promise identical token...