kingjones777/Qwen3.8-Flash-Next-Uncensored-ROCmFP4-FAST-GGUF

### 🔧 Runtime: build the ROCmFPX fork below Stock llama.cpp will not load this file. You need both the `qwen4exp` architecture and the ROCmFP4 tensor types in one tree. Upstream `charlie12345/ROCmFPX` has the ROCmFP4 types but not qwen4exp. Our fork has both: [`kingjones30/ROCmFPX`](https://github.com/kingjones30/ROCmFPX) — a fork of charlie12345/ROCmFPX, branch main. ``bash git clone https://github.com/kingjones30/ROCmFPX.git cd ROCmFPX cmake -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release cmake --build build --target llama-server llama-quantize -j$(nproc) ` ⚠️ **Apply the bundled fix patches before cmake:** qwen4exp-qsa-checkpoint-fix.patch always, plus qwen4exp-mtp-graph-fork.patch if you want --spec-type draft-mtp on this clone. Full steps further down. Verified 2026-08-27 on gfx1151: clean clone → **0 build errors** → llama-server loads a qwen4exp` ROCmFP4 GGUF from this family and generates coherent text.

Qwen3.8-Flash-Next-Uncensored — ROCmFP4 FAST GGUF — AMD Ryzen AI Max+ 395 / gfx1151

⚡ Speculative decoding (MTP) now works — measured +27.7% at short context

The qwen4exp MTP graph shipped with a broken combiner (it mean-pooled the hyper-connection streams), so --spec-type draft-mtp acceptance sat near 0.36 and gave no real speedup. That is now fixed — this repo ships `qwen4exp-mtp-graph.patch`; apply it to the tree the build steps below produce and rebuild (git apply qwen4exp-mtp-graph.patch before cmake --build).

Pair the model with a Flash-Next MTP head from `kingjones777/Qwen3.8-Flash-Next-MTP-Heads-GGUF`. Measured on the Uncensored FAST (imatrix) build with the Q8_0 head (mtp-Qwen3.8-Flash-Next-Q8_0.gguf) at short context (`-c 2048`): acceptance 0.94, 31.80 tok/s vs 24.9 tok/s no-draft (+27.7%), warm 160-token completion, cache_prompt:false. The graph fix and the heads are shared across the Flash-Next family, but this tier's own MTP speed has not been measured, and the Q6_K / Q4 heads were not benchmarked. The head only proposes draft tokens; the main model verifies every one, so your output is unchanged.

llama-server -m <the first shard in this repo>.gguf \
  -md mtp-Qwen3.8-Flash-Next-Q8_0.gguf --spec-type draft-mtp \
  --spec-draft-n-min 0 --spec-draft-n-max 1 --n-gpu-layers-draft 99 \
  -ngl 999 -fa on -np 1 -c 32768 --jinja

-np 1 is required with draft-mtp.

⚠️ Updated 2026-09-17 — re-download if you pulled it earlier. qwen4exp-mtp-graph.patch now carries the models.h and llama-model.cpp hunks it needs. The previous version applied cleanly but failed to compile ('graph_mtp' was not declared in this scope). The bundled patch matches the build steps on this card; for the other build path use qwen4exp-mtp-graph-fork.patch (if you build from a kingjones30/ROCmFPX clone), also bundled here.

Measured plain vs `draft-mtp` — median of 3 per cell, one binary, greedy, cache_prompt:false, 256 generated tokens, -c 2048, Q80 head, Uncensored STRIXLEAN-imatrix weights, gfx1151 / ROCm 7.2.4 (2026-09-17):

⭐ Use `--spec-draft-n-max 1`. It did not lose a single workload here, and it wins most where the next token is predictable. n-max 4 pays for four draft forward passes per step, so it only wins when acceptance is high (reasoning, JSON) and is a genuine loss on code and long-document work. MTP also costs prefill speed, because the draft head processes the prompt too. The older +27.7% figure came from one reasoning-shaped prompt — it holds for that shape, not universally, so measure your own.

✅ Depth: with the bundled checkpoint fix applied, draft-mtp is verified from 2K to 128K — see the box further down for what was measured and what is still open.

⚠️ Research artifact. Refusal behaviour has been removed. This does not add capability — it removes guardrails. Use it deliberately, in a context where that is appropriate, and own the output.

### ✅ Depth: draft-mtp is fixed and measured (2026-09-17) The ≥64K wedge came from context-checkpoint restores leaving the QSA indexer cache (mem_idx) out of the checkpoint. The fix ships here as `qwen4exp-qsa-checkpoint-fix.patch` — it overrides state_write / state_read on llama_memory_hybrid_idx. Apply it with the build steps on this card even if you never use speculative decoding. With it applied, --spec-type draft-mtp ran clean from 2K to 128K on gfx1151: 8 depth rungs, 864 context-checkpoint restores (2 of them prompt-cache rollbacks at 64K), 0 GPU faults, coherent output at every depth. Measured 2026-09-17 on Ryzen AI MAX+ 395 / ROCm 7.2.4 with the Uncensored STRIXLEAN-imatrix weights + `mtp-Qwen3.8-Flash-Next-Q80.gguf, -c 262144, --spec-draft-n-max 4, default context checkpoints. That 128K run used my own fork tree; the exact build steps on this card were verified to 16K. ⚠️ **Still open:** --spec-type ngram-mod at ≥64K has **not** been retested with the patch — the original field report ([…-STRIX-GGUF#6](https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF/discussions/6), thanks [@liusecret](https://huggingface.co/liusecret)) was ngram-mod, so keep -ctxcp 0 -cpent -1 when you use it. And do not use speculative decoding of any kind on Vulkan/gfx1151 — acceptance collapses to 0. A speculative replay stalled` warning on ~2% of restores is expected and harmless: that is the server's livelock guard dropping one draft and decoding that token normally.

Quantized from the BF16 weights published by [orcarouter/Qwen3.8-Flash-Next-Uncensored](https://huggingface.co/orcarouter/Qwen3.8-Flash-Next-Uncensored) — the abliteration work here is theirs, not mine. Go star their repo.

FAST is the smallest tier and the one to take if you are disk-constrained: the Q4_0_ROCMFP4_FAST recipe — attention, experts, token embeddings and the PLE table all ROCmFP4, with only the output head lifted to Q6_K. Converted to BF16 GGUF and quantized by me from their release. 4.27 bpw, 87.94 GiB.

The size matches my aligned build of the same tier to 0.01 GiB — the abliterated checkpoint is structurally identical, so the quant recipe transfers exactly.

The Q6_K head

output.weight is Q6K, never 4-bit. Every sampled token passes through the lm head, so its quantization error lands directly in the argmax. Verified by **exact tensor name** after both quantize and split — `output.weight` is a substring of `attnoutput.weight`, so a loose check reports success on a 4-bit head.

Building a runtime that loads these files

Needs two things in one tree: the qwen4exp architecture and the ROCmFP4 tensor types. charlie12345/ROCmFPX has the ROCmFP4 types but not qwen4exp; the upstream qwen4exp work has no ROCmFP4. The patch combining them ships in this repo: `qwen4exp-on-rocmfpx-d3ca537.patch` (156 KB, 25 files).

git clone https://github.com/charlie12345/ROCmFPX.git
cd ROCmFPX && git checkout d3ca537
curl -LO https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-Uncensored-ROCmFP4-STRIX_LEAN-GGUF/resolve/main/qwen4exp-on-rocmfpx-d3ca537.patch
git apply qwen4exp-on-rocmfpx-d3ca537.patch
# both fixes ship in this repo — apply them before configuring:
curl -LO https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-Uncensored-ROCmFP4-FAST-GGUF/resolve/main/qwen4exp-qsa-checkpoint-fix.patch
git apply qwen4exp-qsa-checkpoint-fix.patch      # checkpoint safety at >=64K: apply this always
curl -LO https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-Uncensored-ROCmFP4-FAST-GGUF/resolve/main/qwen4exp-mtp-graph.patch
git apply qwen4exp-mtp-graph.patch               # only if you want --spec-type draft-mtp
cmake -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --target llama-server llama-quantize -j$(nproc)

Verified from a clean clone: applies without conflicts, compiles with zero errors, and the built llama-server loads these GGUFs and generates. The patch's new files — src/llama-memory-hybrid-idx.{cpp,h} (the QSA indexer's own memory class), src/models/qwen4exp.cpp, conversion/qwen4exp.py — are the pieces hand-copying misses.

Measured — Ryzen AI MAX+ 395, gfx1151, ROCm 7.2.4, full 49/49 offload

• —generation: 22.75 tok/s
• —prompt processing: 387.3 tok/s
• —GPU memory: 63.3 GiB resident — identical to the aligned build

GPU-only, full offload. I do not publish partial-offload speeds.

Measured with one fixed 6,963-token prompt reused across samples (`cache_prompt: false`), run 1 discarded as warm-up, median of the 4 settled samples — spread 1.6 tok/s. An earlier figure of 222 tok/s came from a flawed method that used a different corpus slice per sample; that injected slice-to-slice variance straight into the number. Same file, same GTT (63.6 GiB) — only the measurement changed.

Long context

This model's native max is 262,144, and it runs there on a 128 GB box:

The context window is nearly free — GTT grows only ~4 GiB from 8k to 128k, because Qwen Sparse Attention caps KV. What you pay for is depth: a 200k-token prompt halves generation. It degrades smoothly rather than falling off a cliff.

Refusal / quality (counts only)

Aligned build vs this one, same prompts, greedy, same harness:

Quality is unchanged to the specific failing question, which is the point: the abliteration flipped refusal without the quant damaging the model. Prompts and completions are not published.

Files

Sharded to stay under HF's 50 GB limit. Point --model at the first shard.

Usage

llama-server \
  --model Qwen3.8-Flash-Next-Uncensored-Q4_0-ROCmFP4-FAST-00001-of-00003.gguf \
  --mmproj mmproj-Qwen3.8-Flash-Next-Uncensored-BF16.gguf \
  --host 127.0.0.1 --port 8080 \
  --n-gpu-layers 999 --flash-attn on --fit off \
  --ctx-size 131072 --threads 16 --jinja

Do not use `--no-mmap`. The PLE table is streamed from the file through the page cache; forcing it into anonymous memory gets the process OOM-killed with nothing in the server log.

<!-- CREDITS:START -->

Acknowledgements

[charlie12345/ROCmFPX](https://github.com/charlie12345/ROCmFPX) — defines the ROCmFP4 tensor formats. Every file here was produced with its llama-quantize and runs on its runtime. MIT, based on upstream llama.cpp. The `qwen4exp` architecture is not part of that fork — it comes from upstream llama.cpp work and is applied on top via `qwen4exp-on-rocmfpx-d3ca537.patch` in this repo.

[llama.cpp](https://github.com/ggml-org/llama.cpp) — ggml-org and contributors — the engine, GGUF format and conversion tooling this is built on.

AMD ROCm — the compute platform targeted here (ROCm 7.2.4, gfx1151).

[orcarouter](https://huggingface.co/orcarouter) — published the uncensored BF16 checkpoint this is built from. The abliteration is their engineering; I only converted and quantized it.

Qwen team — the original base model. See base_model; license qwen-community-1.0.

<!-- CREDITS:END -->