hstolte/Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST-GGUF

Qwen 3.8 Next Flash ROCMFPX MUTANT RAZERFAST

[image]

A role-aware, calibration-selected 4-bit + 3-bit build of Qwen3.8-Flash-Next for AMD Strix Halo, using three new GGML weight formats that beat their ROCmFPX / ROCmFP4 predecessors on reconstruction error at exactly the same bits per weight.

125B total parameters, 6B active, 51B n-gram (PLE) embedding table, matched 4B MTP draft head. 90.1 GB target + 2.75 GB MTP sidecar. Loads and generates on a single Ryzen AI MAX+ 395 / Radeon 8060S with the PLE table either resident in shared memory or paged from NVMe with O_DIRECT.

[!IMPORTANT] This is not a stock-GGUF release. Stock llama.cpp will refuse these files with unknown type. You need the matching runtime: https://github.com/hstolte11-collab/ROCmFPX/tree/mutant-razerfast-b10752 (upstream llama.cpp b10752 + a 68-path overlay). Build notes are in `MUTANT-RAZERFAST.md` on that branch.

Files

SHA256SUMS.txt in this repo carries the same two lines. The MTP file is a draft head, not a model; it only works together with the target and is bound to it by shared token_embd.weight / output.weight payloads.

Why "MUTANT"

The three formats below are mutations of the ROCmFPX family: same block geometry, same bits per weight, same 17-byte or 70-byte wire, but the metadata byte is re-purposed to buy back precision that the incumbent layouts throw away.

Q4_RZ5_FAST (GGML type 69) - 4.25 bpw, 770 body matrices

ROCmFP4 FAST stores 32 weights as 16 nibble bytes + 1 metadata byte. The incumbent Codebook10 layout wastes one of its sixteen codes on a duplicate zero (+0 and -0). RZ5 turns that dead code into a +5 or -5 level and uses the top metadata bit to choose the polarity per block, while the lower seven bits keep the UE4M3 scale:

Codebook10 : {0, 1, 2, 3, 4, 6, 8, 10,  0, -1, -2, -3, -4, -6, -8, -10}
RZ5 mode 0 : {0, 1, 2, 3, 4, 6, 8, 10, +5, -1, -2, -3, -4, -6, -8, -10}
RZ5 mode 1 : {0, 1, 2, 3, 4, 6, 8, 10, -5, -1, -2, -3, -4, -6, -8, -10}

Equal wire, one extra usable level, exhaustive per-block scale x mode search.

Q4_UE4M4_FAST (GGML type 71) - 4.25 bpw, 38 matrices

Same 17-byte block and Codebook10 codes as ROCmFP4 FAST, but the full 8-bit metadata byte is an unsigned UE4M4 scale: 4 exponent bits, 4 mantissa bits, all 256 encodings finite, exhaustive 256-way scale search. Finer scale resolution, no extra level.

Q3_PLE_K160_RZ_S3 (GGML type 111) - 3.5 bpw, the 22.4 GB PLE table

per_layer_token_embd.weight is [160, 320001536] - 51.2 billion values. ROCmFP3 packs 3-bit codes {0, +1, +2, +4, -0, -1, -2, -4} with a UE4M3 scale per 16 values. RZ-S3 replaces the duplicate -0 code with +3 or -3 (polarity chosen by the metadata high bit per group). Rows are stored as a row-native 160-value record: 60 code bytes + 10 group metadata bytes = 70 bytes, no padding, so the HIP GET_ROWS kernel can gather one hashed n-gram row in one contiguous read.

Measured: equal-BPW reconstruction precision

These are calibration-set reconstruction metrics on the real BF16 source tensors, computed on CPU before quantization. They are not perplexity and not task scores. They compare candidates at identical bits per weight, so every improvement below is precision gained for free.

Body formats at 4.25 bpw (806 sampled tensor entries, 698,482,688 sampled values per candidate)

Reference = ordinary ROCmFP4 FAST (q4_c10_int4_fast). Lower MSE is better; percentages are error reductions relative to the reference.

RZ5 won all three objectives at the model level and in 22 of 23 tensor roles. The one exception is blk.*.ssm_beta.weight (36 tensors), where UE4M4 wins the two activation-weighted objectives (-6.34% vs -4.16% for RZ5 on normalized activation MSE). So the recipe is role-aware: RZ5 everywhere, UE4M4 on `ssm_beta` and the two output hyper-connection matrices.

Per-role RZ5 reductions on normalized activation MSE range from -2.99% (ffn_gate_exps) to -10.80% (hc_attn_down).

PLE format at 3.5 bpw (1,024-row stratified diagnostic, 41,962 gathers, 16 heads)

Reference = incumbent ROCmFP3 (q3_ue4m3, 70-byte K160 rows). All five candidates have identical wire size.

RZ-S3 also wins unweighted MSE (-27.74%) and count-weighted relative L2 (-15.02%), and takes all 16 per-head wins. It was selected under a frozen objective (count-weighted MSE) before any full-tensor conversion.

Caveat carried verbatim from the screen: bounded 1,024-row diagnostic (32 top-frequency + 32 uniformly sampled rows per head); not an unbiased full-table metric.

Composition of the target (1,224 tensors)

MTP sidecar (34 tensors): 3 Q4_RZ5_FAST routed expert matrices, 16 Q4_UE4M4_FAST, 4 Q8_0 (shared embeddings/output/indexer), 11 F32.

Runtime knobs

Everything below is exposed by the mutant-razerfast-b10752 runtime. Axes are independent; combine as needed.

The disk prompt cache stores target KV, draft KV, and speculative pending_h state together, rejects symlinked components, and on any post-mutation restore failure clears all three before falling back to normal prefill. It never mixes state.

Reference launch (Strix Halo, ROCm 10)

export LD_LIBRARY_PATH=$PWD/build-rocm/bin:/opt/rocm/lib:/opt/rocm/lib/llvm/lib
export HSA_ENABLE_SDMA=0

build-rocm/bin/llama-server \
  -m Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST.gguf \
  --spec-draft-model Qwen3.8-Next-Flash-ROCmFPX-MUTANT-RAZERFAST-MTP.gguf \
  --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.0 \
  -c 8192 -b 512 -ub 512 -ngl 999 -fa on \
  --cache-type-k q8_0 --cache-type-v q8_0 --kv-unified --cont-batching \
  --ctx-checkpoints 4 --checkpoint-min-step 2048 \
  --ngram-on-disk --ngram-io-threads 64 --ngram-cache 256 --ngram-direct-io \
  --jinja --no-mmap --metrics --host 127.0.0.1 --port 8080

Drop the --ngram-* line if you have the memory to keep the PLE resident. --no-mmap is mandatory on ROCm shared-memory systems.

Measured: it loads and generates

Single-request acceptance smokes on one Ryzen AI MAX+ 395 (Radeon 8060S, 128 GB), ROCm 10.0 / HIP 7.15, native gfx1151, HSA_ENABLE_SDMA=0, cold cache, 48-token prompt, -c 8192 -b 512 -ub 512, q8_0 KV. These are smokes, not benchmarks. One request each, no warmup, no repeats.

MTP drafted 79 tokens and the target accepted 43 (54%) on this 64-token greedy generation. Wall-clock speedup from MTP was not demonstrated in these smokes; decode tok/s above counts accepted tokens per second of the whole loop and is lower with MTP on. Whether MTP wins depends on prompt, sampling, and depth - measure it on your workload.

RAM prompt cache, exact-hit replay (target only, resident PLE): 1,544-token prime at 196.61 tok/s prefill; replay of the same prefix processed prompt_n=4 with cache_n=1540 - the cache actually reused 1,540 tokens.

Live memory fit for the disk-PLE, target-only, 1 x 8192 configuration: 65.64 GiB weights + 0.25 GiB PLE row cache + ~4 GiB runtime = ~70 GiB resident, on a host with 112.6 GiB available. A separate first-load run with target + MTP

• —disk PLE peaked at 70.0 GB GTT and loaded in 117.6 s.

What has NOT been measured

Read this before you cite anything above.

• —No perplexity, no benchmark suite, no task accuracy. The quantization claims are reconstruction error on calibration tensors. They say the shipped formats reproduce the original weights more faithfully than the incumbents at equal size. They do not say how the model scores on anything.
• —No sustained throughput benchmark, no tail latency, no batch or -ub sweep, no long-context run beyond 8192.
• —No dense-vs-QSA draft A/B. Dense is the default because two independent published runtimes use it; QSA is exposed as an opt-in experiment.
• —No hybrid or disk-only prompt-cache PASS receipt for this exact package (RAM mode passed; disk mode was interrupted before its receipt).
• —Portability: only ROCm 10 on gfx1151 has run these GGUFs. CPU and Vulkan have reference codecs and no tuned kernels. CUDA compiles the same kernel sources but was not tested with this model.

Provenance

Full lineage, hashes of every input, and the calibration policy identifiers are in `PROVENANCE.md`. Short version:

• —Source weights: Qwen/Qwen3.8-Flash-Next BF16 safetensors (144 files, 360,023,351,514 bytes, size- and header-verified).
• —Runtime: ggml-org/llama.cpp@b96806d96061049a5b574269b049bf6241d63d46 (release b10752) + overlay a96bbaf181adaac2d6b652a109ce7bdf01e73c6f70c2940ab50c6e13a5cd359e (404,588 B).
• —Body-format screen policy a84f7e48...7d8a05; PLE screen policy 936ca2d6...5a8b; final recipe union f85df1cd...181e.
• —Quantized on CPU only, deterministic; header inventories hashed; target and sidecar shared payloads verified byte-identical.

Credits

• —Qwen team for Qwen3.8-Flash-Next and the Qwen Community License 1.0.
• —ggml / llama.cpp authors and contributors, in particular the Qwen3.8-Flash-Next correctness work that landed in b10752: Daniel Han (#27941, and qwen4exp support #27742), Pascal (#28123, #28040, #28023), Jaden_Mach (#27466), itsnotoger (#27991), Sigbjorn Skjaeret (#28159, #28173), JJJYmmm (MTP graph #27739), Ryan Monsurate (NextN draft head #27836).
• —Charlie (`charlie12345`) - ROCmFPX, whose FAST block layout and Codebook10 these formats mutate. https://github.com/charlie12345/ROCmFPX
• —CIRU (`ciru-ai`) for ROCmFP3, the UE4M3 scale codec our PLE rows stay code-compatible with, and the IU4 / PLE-paging / H121 work we studied. https://github.com/ciru-ai/ROCmFPX
• —AgentionAI / Laurent Zuijdwijk for the first public ROCmFP4 FAST Qwen3.8-Flash-Next GGUF + MTP draft head, whose sidecar contract this package follows. https://huggingface.co/agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-GGUF

Hero image generated locally with Z-Image Turbo.

License

Model weights: Qwen Community License 1.0 (copyright Qwen). If you run a Model-as-a-Service or AI Work Assistant business, read clause 2.

Runtime source: MIT, Copyright (c) 2023-2026 The ggml authors, in the GitHub branch.