Qwen3.6-27B family

Evaluation comparison — seven builds, one methodology

What this covers

Credits & thanks

These are other people's models. All I did was measure them. Thanks to everyone below for building and releasing this work.

WikiText-2 perplexity

n_ctx 2048, llama.cpp-compatible · lower is better

Tool-eval final score

full 69 scenarios, --no-think, seed 42, TC-61 excluded

Accuracy benches

--no-think, seed 42 · GSM8K 50q · MMLU 50q · IFEval 20p prompt-level

Multiple-choice accuracy

lm-eval loglikelihood, 0-shot · acc_norm (winogrande: acc)

Logprobs accuracy — raw acc

lm-eval loglikelihood, 0-shot · un-normalized acc per task

Codeneedle positional recall

verbatim function recall under 10K–80K-token contexts

Tool-eval Hard Mode (Category P) — thinking-ON

TC-70–84, seed 42 · not comparable to the --no-think tool-eval scores

Decode throughput — ISL 128

tok/s output vs concurrency, OSL 128, saturation

Decode throughput — ISL 512

tok/s output vs concurrency, OSL 128, saturation

Prefill speed — ISL 128

per-request prefill tok/s (ISL ÷ mean TTFT) vs concurrency

Prefill speed — ISL 512

per-request prefill tok/s (ISL ÷ mean TTFT) vs concurrency

All numbers

All builds serve on the tcclaviger/vllm:latest image. Base Qwen3.6-27B figures are the 2026-07-12 re-measurement; Tess/Aeon/ThinkingCap/35B are their 2026-07-11/12 runs; both 711 runs are 2026-07-25.

Credits: codeneedle (positional recall) originally by Alexander Ziskind, expanded test suite by tcclaviger (Rob Smith). Tool-calling scenarios run on tool-eval-bench by SeraphimSerapis (Tim Messerschmidt), scenario methodology adapted from ToolCall-15 by stevibe. GSM8K, MMLU, IFEval via tool-eval-bench built-in benches at their defaults.