parsimo2010/Qwen3.8-27B-Q4_0_ROCMFP4_STRIX-GGUF

Qwen3.8-27B — Q40ROCMFP4_STRIX (ROCmFPX)

A 4.39 BPW quantization of Qwen3.8-27B using the ROCmFP4 data type family and an unsloth importance matrix, built for AMD Strix Halo (Ryzen AI MAX 300-series / Radeon 8000S, gfx1151).

The point of this file is that it gets a 27B model into 13.98 GiB while keeping the built-in MTP (NextN) speculative-decoding head — so it runs at roughly 1.8× the decode speed of the same file with speculation off, with no separate draft model.

⚠️ This is not a normal GGUF. It uses data types that stock llama.cpp cannot read at all, and it needs a specific fork to load. See Requirements. It will not work in LM Studio, Ollama, or llama.cpp built from upstream master.

Table of contents

• —What is ROCmFP4 / ROCmFPX?
• —How this model was made
• —Requirements: you need a specific llama.cpp fork
• —Prebuilt Windows binaries (no build required)
• —Build and run
• —Recommended settings (validated on Strix Halo)
• —How the settings were chosen (full sweep results)
• —KV cache options
• —Prompt cache sizing
• —Known limitations
• —Reproducing this exact file
• —Credits

What is ROCmFP4 / ROCmFPX?

ROCmFP4 is a 4-bit quantization family with custom GPU kernels written for AMD hardware (ROCMFP4, ROCMFP4_FAST, and a STRIX mix preset), added by the charlie12345/ROCmFPX fork of llama.cpp. The same fork also adds other custom types (ROCmFPX 2/3/6/8-bit, TurboQuant).

Two things matter to a user:

1. 1.They are not in upstream `llama.cpp`. Upstream has no GGML_TYPE_Q4_0_ROCMFP4; loading this file with a stock build fails. A GGUF carrying these types is only readable by this fork (or a build that has merged these kernels).
2. 2.`_STRIX` is a recipe, not a single type. It is a per-tensor assignment tuned for Strix Halo's memory bandwidth: hot weights get the faster variant, attention K/V get the higher-quality dual-scale variant, and embeddings/norms are kept in higher precision. See the type map below.

How this model was made

1. 1.Downloaded the BF16 GGUF of Qwen3.8-27B from unsloth (2 shards, 50.9 GiB total) plus unsloth's importance matrix.
2. 2.Re-quantized with the ROCmFP4 `_STRIX` recipe, driven by that imatrix.
3. 3.Left the source split intact — no --keep-split — so the two shards merged into a single output file, which is what keeps the MTP head in the same file as the model.

The imatrix is genuinely consumed, not decoration: both ROCmFP4 quantizers are imatrix-aware (rocmfp4_quantize_q4_0_weighted / ..._fast_weighted) and use it for imatrix-weighted scale search.

The exact command:

llama-quantize \
  --imatrix imatrix_unsloth.gguf \
  Qwen3.8-27B-BF16-00001-of-00002.gguf \
  Qwen3.8-27B-Q4_0_ROCMFP4_STRIX.gguf \
  Q4_0_ROCMFP4_STRIX

llama-quantize reports the source as 52115.19 MiB @ 16.00 BPW and the output as 14307.95 MiB @ 4.39 BPW, with zero shape fallbacks.

The resulting type map

The MTP head is preserved

Qwen3.8-27B ships a NextN/MTP head as blk.64.nextn.*. It survives quantization and is present in this file — you do not need a separate draft model:

qwen35.nextn_predict_layers = 1
Q8_0  blk.64.nextn.eh_proj.weight
F32   blk.64.nextn.enorm.weight / hnorm.weight / shared_head_norm.weight

When served correctly, the log confirms it is drafting from the model's own head:

load_model: creating MTP draft context against the target model '...ROCMFP4_STRIX.gguf'
common_speculative_state_draft_mtp: - n_max=4, n_min=0, p_min=0.00, n_embd=5120
load_model: speculative decoding context initialized

Requirements: you need a specific llama.cpp fork

This file will not load in upstream `llama.cpp`. Use the fork it was made with:

• —Repo: https://github.com/charlie12345/ROCmFPX
• —Validated commit: `c49ebdbd5c9f01ec242369f9e7f7967855f80cba` (branch main)

Because forks move, pin a commit. Q4_0_ROCMFP4 support is a prerequisite; check that llama-quantize --help lists Q4_0_ROCMFP4_STRIX (type 105) before blaming the file.

Prebuilt Windows binaries (no build required)

A prebuilt Windows x64 build for gfx1151 is attached to this repo, so you can skip compiling:

[`llama.cpp-ROCmFPX-windows-gfx1151.zip`](./llama.cpp-ROCmFPX-windows-gfx1151.zip) — 58 MB

It contains llama-server, llama-cli, llama-quantize, llama-imatrix, llama-bench, llama-perplexity and test-backend-ops, plus every matching DLL (ggml-hip, ggml-vulkan, ggml-cpu, ggml-base, llama, llama-common, mtmd). Built from commit c49ebdbd… with the flags below under MSVC 14.44. Extract it anywhere and run the executables from that folder — the DLLs must stay next to the .exe files.

The archive ships only this project's binaries. Install the runtime parts yourself:

Also install the Visual C++ 2015–2022 x64 Redistributable (MSVCP140.dll, VCRUNTIME140.dll), which almost every Windows machine already has.

The 7 in amdhip64_7.dll means the ROCm major version must be 7.x, not 6.x. Device code for gfx1151 is baked into ggml-hip.dll, so the ROCm backend will not run on a different AMD GPU — use -dev Vulkan0 there, or build from source.

SHA256SUMS inside the archive lists a checksum for every file (Get-FileHash on Windows, sha256sum -c elsewhere). Because the zip contains .exe files fetched from the internet, Windows may mark them as blocked; if they refuse to launch, right-click the zip → Properties → tick Unblock → OK, and then extract.

Build notes

You need an HIP-capable build (-DGGML_HIP=ON). A Vulkan-only build is much slower on prefill.

cmake -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_HIP=ON \
  -DGGML_HIP_FORCE_MMQ=ON \
  -DGGML_HIP_ROCWMMA_FATTN=OFF \
  -DCMAKE_HIP_ARCHITECTURES=gfx1151 \
  -DGGML_VULKAN=ON \
  -DLLAMA_BUILD_SERVER=ON
cmake --build build -j

-DGGML_HIP_FORCE_MMQ=ON is required by the ROCmFP4 kernels — do not drop it. gfx1151 is Strix Halo; substitute your target.

Two gotchas found while building this on Windows, in case they save someone an afternoon:

• —Pin MSVC to 14.44. On MSVC 14.51+ a <cmath> change makes ROCm clang redeclare isgreater/isless/… as __device__, and every ggml-cuda/ translation unit fails. Note that vswhere -latest may return an install whose only toolset is the broken one.
• —Pass absolute paths to the HIP SDK's `clang.exe`/`clang++.exe` so a different clang earlier on PATH cannot be picked up.

Build and run

Runtime DLLs/SOs come from the HIP SDK (ROCm 7.1 here), so put them on the library path first. On Windows you can skip this section entirely by using the prebuilt zip.

Tuned command (the one validated below):

llama-server \
  -m Qwen3.8-27B-Q4_0_ROCMFP4_STRIX.gguf \
  -dev ROCm0 -ngl 999 -fa on --jinja \
  -c 262144 -b 512 -ub 512 -t 16 \
  -ctk q8_0 -ctv q8_0 --cache-ram 10240 \
  --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.0 \
  --host 127.0.0.1 --port 8090

The chat UI is embedded in llama-server — open http://127.0.0.1:8090.

For a one-shot CLI check (note -st, not -no-cnv — this fork's llama-cli rejects -no-cnv / --no-conversation):

llama-cli -m Qwen3.8-27B-Q4_0_ROCMFP4_STRIX.gguf \
  -dev ROCm0 -ngl 999 -fa on --jinja -st \
  -p "Write a Python LRU cache with O(1) get/put." -n 256 \
  --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.0

Recommended settings (validated on Strix Halo)

Test machine: AMD Ryzen AI MAX+ 395, Radeon 8060S (gfx1151), 128 GB unified LPDDR5X, Windows 11, ROCm 7.1 HIP SDK.

All figures are greedy decoding on this machine. Prefill is dominated by context depth, decode by memory bandwidth; both scale with your hardware.

Why each setting

Two of these are the opposite of what you'd guess, so they are worth repeating: the default `p-min` of 0.75 costs ~12 %, and `n-max 6` costs 15–25 %. Both are the "more conservative" direction, and both are slower.

How the settings were chosen (full sweep results)

Everything below was swept on the test machine with greedy sampling. If you only read one thing, read the two "wrong defaults" above.

Draft depth — --spec-draft-n-max (prose)

Drafts beyond position ~3 are almost never accepted (acceptance by position: 0.747, 0.467, 0.267, 0.120, 0.053, 0.000), yet the target still pays a batched verification over all 6 candidates plus extra recurrent-state rollback. On agent-style prompts, 3 vs 4 is a wash that depends on output type — n-max 3 favours code emission, n-max 4 favours thinking-heavy output:

Since a reasoning model emits far more thinking tokens than answer tokens, 4 is the better default.

Draft early-stop — --spec-draft-p-min

Monotonic — raising p-min always costs speed. Note the misleading acceptance column: p-min 0.9 has the highest acceptance fraction and is 22 % slower, because it stops drafting early and so converts fewer tokens per verification step. Judge speculative decoding by tokens per step, not by acceptance rate. p-min is a draft early-stop only — every accepted token is still verified against the target model, so it cannot affect output quality.

Backend — -dev ROCm0 vs -dev Vulkan0

Prefill (ROCm wins):

Decode (Vulkan wins, until MTP is enabled):

So Vulkan's large raw-decode advantage essentially disappears once MTP is on. Total-time model for a turn at 64k context (1200 tokens prefilled, 900 generated), where 1/K = a full 64k re-prefill every K turns:

Crossover is roughly one full re-prefill per 25–30 turns. ROCm0 is the default because its downside is small (+6 % if you never miss) and its upside is large (−15 % when you do). Choose Vulkan0 only if your prefix is append-only and your turns are decode-bound.

Batch size — -b / -ub

No-MTP llama-bench (pp512 / pp4096 / tg128):

Use -ub 512.

KV cache options

The fork accepts these KV cache types (-ctk / -ctv): f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1, plus fork-only q4_0_rocmfp4, q4_0_rocmfp4_fast, q3_0_rocmfpx, q6_0_rocmfpx, q8_0_rocmfpx, and TurboQuant turbo3, turbo4. Flash attention is enabled for all quantized KV types in this build.

Measured at 128k context (RSS = whole server working set; decode relative to f16):

Read this before picking a KV type:

• —The fork's ROCmFPX fp cache types (`q8_0_rocmfpx`, `q6_0_rocmfpx`) are 2–3× slower — 34–43 % of f16 decode on Vulkan. Despite the name matching the model's family, they are the wrong choice for the cache. Only the ROCmFP4 pair has a fast path on Vulkan.
• —`turbo4` (TurboQuant) is the best fork KV type — the fastest decode measured (106 % of f16) while saving 5.7 GB, and near-free on ROCm.
• —`q8_0` is the safe default and is what the recommended config uses: ~100 % of f16 speed, best Vulkan prefill, saves 3.7 GB. 8-bit KV is as close to lossless as quantized KV gets.
• —KV quantization generally costs a little decode speed (dequant work) — it buys memory, not speed, except turbo4 on Vulkan.
• —The MTP draft KV is always `f16` regardless of -ctk/-ctv (the draft context logs cache_k=f16, cache_v=f16), so it adds a fixed ~4 KiB/token that you cannot quantize this way.
• —The fork applies a Walsh–Hadamard rotation to the KV cache for accuracy, which is disabled for fp3/TurboQuant types (they carry their own), logged as attention rotation disabled for fp3 ROCmFPX/TurboQuant KV cache.

Naming trap: TurboQuant's CLI names are turbo3 / turbo4, not turbo3_0 / turbo4_0. The latter fails with Unsupported cache type even though GGML_TYPE_TURBO4_0 is in the allow-list.

Prompt cache sizing

llama-server can keep conversation prefixes resident in RAM (--cache-ram) so a returning agent skips re-prefill. Measured state size (q8_0 KV), from the prompt_save: ... total state size log line:

Perfectly linear: state(N) = 149.6 MiB + N × 0.037148 MiB (38.04 KiB/token). That checks out against the architecture exactly — main KV is 16 layers × 4 KV heads × 256 dim × 2 = 32,768 elem/token × 1.0625 B = 34.0 KiB, plus 4.0 KiB f16 draft.

So `--cache-ram 10240` fits a whole 262k conversation with ~5 % headroom. The ~150 MiB fixed term is the hybrid model's recurrent state per cached conversation, so each additional cached 262k conversation needs another ~9.7 GiB. The stock default is 8192 MiB, which is not enough for a full 262k context.

Known limitations

• —Requires the fork. Will not load in upstream llama.cpp, LM Studio, or Ollama. See Requirements.
• —MTP output is not bit-identical to non-speculative greedy decoding. In a controlled A/B (same server, same prompt, greedy, only MTP toggled) output diverged — an equally coherent but differently-phrased continuation, at character 503 of ~800. This persisted even with the fork's --spec-mtp-strict-qwen (which requires -np 1 and claims exact greedy output). Likely cause is batched verification shifting logits enough to flip a near-tie; this is a hybrid SSM/attention model, which is the hardest case for state rollback. Treat MTP as a speed feature, not a reproducible-output feature. If you need reproducible greedy output, disable speculation.
• —KV-cache quantization quality was not measured. The speed and memory figures above say nothing about accuracy. q8_0 was chosen as the default precisely because it is near-lossless and costs nothing measurable; turbo4 is faster but is 4-bit. This has not been validated with perplexity or task evals.
• —Quantization quality of the model itself was not independently evaluated (no perplexity or benchmark run). The type map follows the _STRIX recipe and calibration uses unsloth's own imatrix, but treat quality as unverified until you test your own workload.
• —One unreproduced GPU fault was observed: a single FLASH_ATTN_EXT failed / ROCm error: unspecified launch failure on a 64k prefill with --cache-ram 65536 and three prompts already cached. f16, q8_0, and turbo4 subsequently each prefilled 63,813 tokens cleanly, so it looks transient rather than type-specific. Related gotcha: a crashed HIP context leaves the GPU wedged, and the next launch dies ~0.2 s in at `hipMemGetInfo`. That second failure is not a new bug — kill the process, wait ~30 s, relaunch.
• —Model metadata says `general.quantized_by = "Unsloth"`, inherited from unsloth's BF16 source file. The ROCmFP4 quantization was performed locally, so correct that field (--override-kv general.quantized_by=str:<you>) if you redistribute.

Reproducing this exact file

1. 1.Get the fork at the pinned commit and build with -DGGML_HIP=ON -DGGML_HIP_FORCE_MMQ=ON (see Build notes).
2. 2.Download Qwen3.8-27B-BF16-00001-of-00002.gguf, ...-00002-of-00002.gguf and imatrix_unsloth.gguf from unsloth/Qwen3.8-27B-GGUF.
3. 3.Quantize — pass only shard 1 as input and omit `--keep-split`, so the shards merge into one file that retains the MTP head:

llama-quantize \
  --imatrix imatrix_unsloth.gguf \
  Qwen3.8-27B-BF16-00001-of-00002.gguf \
  Qwen3.8-27B-Q4_0_ROCMFP4_STRIX.gguf \
  Q4_0_ROCMFP4_STRIX 32

Use --dry-run first: it prints the projected size/BPW and the full per-tensor type map without writing anything. Expect 14307.95 MiB @ 4.39 BPW and 0 fallbacks.

1. 1.Verify the output carries the MTP head — blk.64.nextn.* tensors (4 of them) and qwen35.nextn_predict_layers = 1.

Credits

• —Base model: Qwen/Qwen3.8-27B (Apache-2.0)
• —BF16 GGUF + importance matrix: unsloth — the imatrix does real work here (imatrix-weighted scale search), so this quantization would be worse without it.
• —ROCmFP4 / ROCmFPX types and kernels: charlie12345/ROCmFPX — the fork that makes these types exist, and the only way to run this file.
• —Upstream: ggml-org/llama.cpp

All benchmark numbers in this card were measured on the single Strix Halo machine described above, with greedy decoding. They will differ on other hardware; the relative comparisons (ROCm vs Vulkan, n-max 4 vs 6, p-min 0.0 vs 0.75, -ub 512 vs 2048) are the parts worth carrying over.