メインコンテンツへスキップ
見出し画像

Qwen3.8-27B をとりあえず動かす- Q4: 70t/s, Q2: 100t/s

    所感

    良い感じ。Unsloth は、本家と同時に MTP 対応で GGUF を公開してくれた。

    『RTX5070Ti + RTX5070』と『RTX5070Ti』の 2 パターンで動かす。

    ただ、ワンショットは凄いが、エージェンティックな挙動は 3.6 の方が良い、チャットテンプレートが良くないとか、言われている。まー、初期不具合(かどうかは分からない)的なやつは良くあること。

    のんびり遊びながら、動向をチェックしていく。
    今回は、動かすだけ。

    UD-Q4_K_XL (VRAM: 28 GB)

    RTX5070Ti + RTX5070 / Context Length: 128K

    llama-server -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL \
      -ngl 99 -fa on \
      --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
      --presence-penalty 0.0 --repeat-penalty 1.0 \
      --jinja --chat-template-kwargs '{"reasoning_effort": "low"}' \
      -cb -np 1 \
      --spec-type draft-mtp --spec-draft-n-max 2 \
      -ts 18,12 -c 131072 -ctk q8_0 -ctv q8_0

    nvidia-smi

    Sat Aug 15 10:56:44 2026
    +-----------------------------------------------------------------------------------------+
    | NVIDIA-SMI 610.43.02              KMD Version: 610.43.02     CUDA UMD Version: 13.3     |
    +-----------------------------------------+------------------------+----------------------+
    | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
    |                                         |                        |               MIG M. |
    |=========================================+========================+======================|
    |   0  NVIDIA GeForce RTX 5070 Ti     On  |   00000000:01:00.0 Off |                  N/A |
    |  0%   44C    P8             11W /  300W |   15043MiB /  16303MiB |      0%      Default |
    |                                         |                        |                  N/A |
    +-----------------------------------------+------------------------+----------------------+
    |   1  NVIDIA GeForce RTX 5070        On  |   00000000:04:00.0 Off |                  N/A |
    |  0%   40C    P8              3W /  250W |   11361MiB /  12227MiB |      0%      Default |
    |                                         |                        |                  N/A |
    +-----------------------------------------+------------------------+----------------------+
    
    +-----------------------------------------------------------------------------------------+
    | Processes:                                                                              |
    |  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
    |        ID   ID                                                               Usage      |
    |=========================================================================================|
    |    0   N/A  N/A            3449      G   /usr/lib/xorg/Xorg                        4MiB |
    |    0   N/A  N/A           22038      C   llama-server                          15018MiB |
    |    1   N/A  N/A            3449      G   /usr/lib/xorg/Xorg                        4MiB |
    |    1   N/A  N/A           22038      C   llama-server                          11336MiB |
    +-----------------------------------------------------------------------------------------+

    設計

    • Prompt: 494.53 tokens per second

    • Generate: 60.79 tokens per second

    1.31.346.000 I slot print_timing: id  0 | task 0 | prompt eval time =     333.65 ms /   165 tokens (    2.02 ms per token,   494.53 tokens per second)
    1.31.346.002 I slot print_timing: id  0 | task 0 |        eval time =   34906.21 ms /  2123 tokens (   16.45 ms per token,    60.79 tokens per second)
    1.31.346.003 I slot print_timing: id  0 | task 0 |       total time =   35239.86 ms /  2288 tokens
    1.31.346.013 I slot print_timing: id  0 | task 0 |    graphs reused =        969
    1.31.346.015 I slot print_timing: id  0 | task 0 | draft acceptance = 0.58598 ( 1145 accepted /  1954 generated), mean len =  2.17
    1.31.346.075 I slot      release: id  0 | task 0 | stop processing: n_tokens = 2287, truncated = 0

    実装

    • Prompt: 1619.46 tokens per second

    • Generate: 71.33 tokens per second

    9.50.416.527 I slot print_timing: id  0 | task 981 | prompt eval time =    1335.02 ms /  2162 tokens (    0.62 ms per token,  1619.46 tokens per second)
    9.50.416.530 I slot print_timing: id  0 | task 981 |        eval time =  333850.58 ms / 23815 tokens (   14.02 ms per token,    71.33 tokens per second)
    9.50.416.531 I slot print_timing: id  0 | task 981 |       total time =  335185.59 ms / 25977 tokens
    9.50.416.532 I slot print_timing: id  0 | task 981 |    graphs reused =       9539
    9.50.416.534 I slot print_timing: id  0 | task 981 | draft acceptance = 0.87431 (15150 accepted / 17328 generated), mean len =  2.75
    9.50.416.653 I slot      release: id  0 | task 981 | stop processing: n_tokens = 26137, truncated = 0

    生成の雰囲気

    生成されたもの

    UD-Q5_K_XL (VRAM: 28 GB)

    RTX5070Ti + RTX5070 / Context Length: 96K

    llama-server -hf unsloth/Qwen3.8-27B-GGUF:UD-Q5_K_XL \
      -ngl 99 -fa on \
      --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
      --presence-penalty 0.0 --repeat-penalty 1.0 \
      --jinja --chat-template-kwargs '{"reasoning_effort": "low"}' \
      -cb -np 1 \
      --spec-type draft-mtp --spec-draft-n-max 2 \
      -ts 18,12 -c 98304 -ctk q8_0 -ctv q8_0

    設計

    • Prompt: 457.36 tokens per second

    • Generate: 53.35 tokens per second

    2.19.131.193 I slot print_timing: id  0 | task 0 | prompt eval time =     360.77 ms /   165 tokens (    2.19 ms per token,   457.36 tokens per second)
    2.19.131.197 I slot print_timing: id  0 | task 0 |        eval time =   40581.32 ms /  2166 tokens (   18.74 ms per token,    53.35 tokens per second)
    2.19.131.198 I slot print_timing: id  0 | task 0 |       total time =   40942.09 ms /  2331 tokens
    2.19.131.204 I slot print_timing: id  0 | task 0 |    graphs reused =       1001
    2.19.131.206 I slot print_timing: id  0 | task 0 | draft acceptance = 0.57178 ( 1155 accepted /  2020 generated), mean len =  2.14
    2.19.131.268 I slot      release: id  0 | task 0 | stop processing: n_tokens = 2330, truncated = 0

    実装

    • Prompt: 1518.63 tokens per second

    • Generate: 71.28 tokens per second

    5.12.257.069 I slot print_timing: id  0 | task 1014 | prompt eval time =    1447.35 ms /  2198 tokens (    0.66 ms per token,  1518.63 tokens per second)
    5.12.257.072 I slot print_timing: id  0 | task 1014 |        eval time =  108957.92 ms /  7768 tokens (   14.03 ms per token,    71.28 tokens per second)
    5.12.257.073 I slot print_timing: id  0 | task 1014 |       total time =  110405.28 ms /  9966 tokens
    5.12.257.074 I slot print_timing: id  0 | task 1014 |    graphs reused =       3706
    5.12.257.076 I slot print_timing: id  0 | task 1014 | draft acceptance = 0.91941 ( 5031 accepted /  5472 generated), mean len =  2.84
    5.12.257.194 I slot      release: id  0 | task 1014 | stop processing: n_tokens = 10126, truncated = 0

    UD-Q2_K_XL (VRAM: 16 GB)

    RTX5070Ti / Context Length: 64K

    CUDA_VISIBLE_DEVICES=0 \
    llama-server -hf unsloth/Qwen3.8-27B-GGUF:UD-Q2_K_XL \
      -ngl 99 -fa on \
      --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
      --presence-penalty 0.0 --repeat-penalty 1.0 \
      --jinja --chat-template-kwargs '{"reasoning_effort": "low"}' \
      -cb -np 1 \
      --spec-type draft-mtp --spec-draft-n-max 2 \
      -c 65536 -ctk q8_0 -ctv q8_0

    nvidia-smi

    Sat Aug 15 11:19:25 2026
    +-----------------------------------------------------------------------------------------+
    | NVIDIA-SMI 610.43.02              KMD Version: 610.43.02     CUDA UMD Version: 13.3     |
    +-----------------------------------------+------------------------+----------------------+
    | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
    |                                         |                        |               MIG M. |
    |=========================================+========================+======================|
    |   0  NVIDIA GeForce RTX 5070 Ti     On  |   00000000:01:00.0 Off |                  N/A |
    |  0%   43C    P8             14W /  300W |   14709MiB /  16303MiB |      0%      Default |
    |                                         |                        |                  N/A |
    +-----------------------------------------+------------------------+----------------------+
    |   1  NVIDIA GeForce RTX 5070        On  |   00000000:04:00.0 Off |                  N/A |
    |  0%   35C    P8              4W /  250W |      15MiB /  12227MiB |      0%      Default |
    |                                         |                        |                  N/A |
    +-----------------------------------------+------------------------+----------------------+
    
    +-----------------------------------------------------------------------------------------+
    | Processes:                                                                              |
    |  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
    |        ID   ID                                                               Usage      |
    |=========================================================================================|
    |    0   N/A  N/A            3449      G   /usr/lib/xorg/Xorg                        4MiB |
    |    0   N/A  N/A           28245      C   llama-server                          14686MiB |
    |    1   N/A  N/A            3449      G   /usr/lib/xorg/Xorg                        4MiB |
    +-----------------------------------------------------------------------------------------+

    設計

    • Prompt: 547.33 tokens per second

    • Generate: 84.97 tokens per second

    0.47.757.044 I slot print_timing: id  0 | task 0 | prompt eval time =     301.46 ms /   165 tokens (    1.83 ms per token,   547.33 tokens per second)
    0.47.757.047 I slot print_timing: id  0 | task 0 |        eval time =   26104.45 ms /  2219 tokens (   11.77 ms per token,    84.97 tokens per second)
    0.47.757.047 I slot print_timing: id  0 | task 0 |       total time =   26405.92 ms /  2384 tokens
    0.47.757.052 I slot print_timing: id  0 | task 0 |    graphs reused =       1036
    0.47.757.054 I slot print_timing: id  0 | task 0 | draft acceptance = 0.56124 ( 1173 accepted /  2090 generated), mean len =  2.12
    0.47.757.106 I slot      release: id  0 | task 0 | stop processing: n_tokens = 2383, truncated = 0

    実装

    • Prompt: 1580.41 tokens per second

    • Generate: 107.59 tokens per second

    2.11.070.865 I slot print_timing: id  0 | task 1049 | prompt eval time =    1415.46 ms /  2237 tokens (    0.63 ms per token,  1580.41 tokens per second)
    2.11.070.868 I slot print_timing: id  0 | task 1049 |        eval time =   59961.08 ms /  6452 tokens (    9.29 ms per token,   107.59 tokens per second)
    2.11.070.869 I slot print_timing: id  0 | task 1049 |       total time =   61376.53 ms /  8689 tokens
    2.11.070.870 I slot print_timing: id  0 | task 1049 |    graphs reused =       3342
    2.11.070.871 I slot print_timing: id  0 | task 1049 | draft acceptance = 0.88336 ( 4120 accepted /  4664 generated), mean len =  2.77
    2.11.070.977 I slot      release: id  0 | task 1049 | stop processing: n_tokens = 8850, truncated = 0

    生成の雰囲気

    生成されたもの

    ゲームとして面白かったもの

    何回か生成したので、その中で出来が良かったものを記録として。

    Best Game (UD-Q2_K_XL ワンショット)

    Best UI (UD-Q4_K_XL ワンショット)

    (おまけ)Ngram を添えて

    気持ち早くなる(かな?)

    --spec-type draft-mtp,ngram-mod 
    --spec-draft-n-max 6 
    --spec-draft-p-min 0.75 
    --spec-ngram-mod-n-match 24 
    --spec-ngram-mod-n-min 48 
    --spec-ngram-mod-n-max 64 
    --reasoning-budget 8196 
    --chat-template-kwargs "{\"preserve_thinking\": true}"

    How to combine mtp and ngram to get extra performance ? I just use these flags, I have an app to help with that you can use that if you want it's open source.

    --spec-type draft-mtp,ngram-mod
    --spec-draft-n-max 6
    --spec-draft-p-min 0.75
    --spec-ngram-mod-n-match 24… https://t.co/2xuq53dRFA

    — Ark (@quimedesu) August 14, 2026


    あなたへのおすすめ