CoolFace
Modelpublic

petr567/Qwen3.5-2B-MTP-fast-GGUF

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face
1likes168downloads
Model Card

Qwen3.5-2B MTP fast

Fast MTP profiles for Bartowski's Qwen3.5-2B Q4KM GGUF in LM Studio and llama.cpp Vulkan.

The GGUF weights are unchanged. The speed-up is enabled by the included LM Studio MTP load profile.

Files

FileDescription
Qwen_Qwen3.5-2B-Q4_K_M.ggufQ4KM model with built-in MTP heads
mmproj-Qwen_Qwen3.5-2B-f16.ggufVision projector
load_profile.ps1LM Studio baseline/fast profile loader
benchmark-summary.jsonMachine-readable benchmark summary
benchmark-summary.csvCompact context-speed table
cloudpersonal-benchmark-summary.jsonStrix Halo server benchmark summary
cloudpersonal-benchmark.csvStrix Halo context-speed table

LM Studio fast profile

  • —LM Studio 0.4.20 or newer
  • —context: up to 262,144 tokens
  • —parallel: 1
  • —Flash Attention: on
  • —KV cache GPU offload: on
  • —MTP: on
  • —maximum draft tokens: 2
  • —minimum continue probability: 0.20

Start LM Studio's local server on port 1234, then run:

powershell
powershell.exe -NoProfile -ExecutionPolicy Bypass -File .\load_profile.ps1 -Profile ornith -ContextLength 262144

RTX 4060 Laptop measurements

Mean decode speed, three runs per context, 128 generated tokens:

Prompt contextMTP offMTP fastChange
1K128.86 tok/s177.92 tok/s+38.08%
4K126.74 tok/s175.16 tok/s+38.20%
8K122.89 tok/s172.74 tok/s+40.57%
16K117.35 tok/s165.21 tok/s+40.78%
32K106.28 tok/s148.93 tok/s+40.13%
64K88.71 tok/s132.06 tok/s+48.86%
128K67.85 tok/s105.60 tok/s+55.64%
250K48.27 tok/s76.82 tok/s+59.14%
  • —maximum single run: 179.09 tok/s
  • —maximum repeated mean: 177.92 tok/s
  • —geometric mean speed-up over all contexts: 1.4497x
  • —maximum loaded context: 262,144 tokens

Test system: Windows 11, Intel Core i7-13650HX, 63.74 GiB RAM, NVIDIA RTX 4060 Laptop 8 GiB, LM Studio 0.4.20, llama.cpp CUDA runtime 2.27.1.

Ryzen AI MAX+ 395 / Radeon 8060S server measurements

Direct llama.cpp b9994 Vulkan A/B on the same GGUF and prompts. The server used a 262,144-token window, Q8_0 K/V cache, Flash Attention, full GPU offload, parallel 1, and three runs per context. Other resident AI services were idle during the test.

Prompt contextMTP offMTP fastChange
1K125.68 tok/s203.82 tok/s+62.18%
4K123.79 tok/s197.10 tok/s+59.22%
8K121.23 tok/s188.57 tok/s+55.55%
16K115.64 tok/s180.21 tok/s+55.84%
32K108.47 tok/s159.48 tok/s+47.02%
64K94.00 tok/s131.15 tok/s+39.52%
128K75.76 tok/s95.64 tok/s+26.24%
250K56.06 tok/s63.67 tok/s+13.59%
  • —maximum single run: 205.20 tok/s
  • —maximum repeated mean: 203.82 tok/s
  • —geometric mean speed-up over all contexts: 1.4393x
  • —deterministic output hashes matched at every context

Server: AMD Ryzen AI MAX+ 395 with Radeon 8060S, 128 GiB UMA, Ubuntu, llama.cpp b9994 (14d3ba45f), Vulkan.

Credits