kingjones777/KAT-Coder-V2.5-Dev-Abliterated-MTP-ROCmFP4-GGUF

KAT-Coder-V2.5-Dev-Abliterated-MTP — ROCmFP4 for AMD Strix Halo (gfx1151)

The model here is not our work. The abliterated MTP checkpoint is [jakeroxs](https://huggingface.co/jakeroxs)'s (`KAT-Coder-V2.5-Dev-35B-A3B-MTP-ABLITERATED-GGUF`); the base model is [Kwaipilot](https://huggingface.co/Kwaipilot)'s (`KAT-Coder-V2.5-Dev`). This repository adds only the ROCmFP4/ROCmFPX quantisation ladder, the dual-backend build, and the measurements. Go star their repos.

Four quantisations of the abliterated KAT-Coder-V2.5-Dev 35B-A3B MoE, each carrying a working Multi-Token-Prediction block, built for AMD Ryzen AI Max+ 395 / Radeon 8060S. Every file runs on BOTH the HIP (ROCm) and Vulkan backends from a single binary — the backend is a runtime `-dev` flag, not a rebuild.

Lead result: MTP is worth +6.5 % on Vulkan and +8.9 % on ROCm here, and the backend crossover flips by tier — Vulkan takes the 4-bit tiers, ROCm takes both Q6 tiers and runs them in 7.2 GiB less GTT.

✅ This model answers cleanly out of the box. 6/6 non-empty content at every budget from 400 tokens up, on all four workloads. That is worth stating because its architectural sibling does not — see the note at the end.

Which file should I use?

`STRIX_LEAN` (106) on Vulkan is the pick for coding work — fastest on the code workload (80.76 tok/s) and the smallest file. If your work is prompt-heavy, the same file on -dev ROCm0 prefills at 478 tok/s.

⛔ ftype 105 (`STRIX`) is deliberately not published — it leaves output.weight and the *_shexp tensors at 4 bit.

Quick start — the backend is one flag

# Vulkan — best decode (chat, agentic loops)
llama-server -m KAT-Coder-V2.5-Dev-Abliterated-MTP-Q4_0_ROCMFP4_STRIX_LEAN.gguf \
  -dev Vulkan0 -ngl 999 -fa on -fit off -np 1 -c 8192 --jinja \
  --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.0 \
  --host 0.0.0.0 --port 8080

# HIP / ROCm — best prefill (long prompts, cold starts, RAG)
#   SAME FILE. Only -dev changes.
llama-server -m KAT-Coder-V2.5-Dev-Abliterated-MTP-Q4_0_ROCMFP4_STRIX_LEAN.gguf \
  -dev ROCm0 -ngl 999 -fa on -fit off -np 1 -c 8192 --jinja \
  --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.0 \
  --host 0.0.0.0 --port 8080

These MTP flags are for single-shot use. For multi-turn / agentic work drop them entirely — see Multi-turn agentic work below, where speculation costs 31 % end-to-end. For single-shot, `--spec-draft-p-min 0.0` is not optional if you want the speed. The default is 0.75 and it throttles the draft.

⛔ You need a ROCmFPX build with Vulkan enabled — stock llama.cpp will NOT load these files

ROCmFP4/ROCmFPX use ggml tensor types 100–119. Upstream llama.cpp's type table stops at 43, so a stock build rejects these files outright, on any backend:

gguf_init_from_reader: tensor '...' has invalid ggml type 101. should be in [0, 43)

Build one with both backends — the exact configuration behind every number on this card:

git clone https://github.com/charlie12345/ROCmFPX.git && cd ROCmFPX
cmake -B build-hipvk -G "Unix Makefiles" \
  -DCMAKE_MAKE_PROGRAM=/usr/bin/gmake \
  -DCMAKE_C_COMPILER=/usr/bin/cc -DCMAKE_CXX_COMPILER=/usr/bin/c++ \
  -DCMAKE_HIP_COMPILER=/opt/rocm-7.2.4/lib/llvm/bin/clang \
  -DROCM_PATH=/opt/rocm-7.2.4 -DCMAKE_PREFIX_PATH=/opt/rocm-7.2.4 \
  -DGGML_HIP=ON -DGGML_VULKAN=ON -DAMDGPU_TARGETS=gfx1151 \
  -DVulkan_GLSLC_EXECUTABLE=/usr/bin/glslc \
  -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=OFF -DLLAMA_BUILD_WEBUI=OFF
cmake --build build-hipvk -j12
build-hipvk/bin/llama-server --list-devices   # must list ROCm0 AND Vulkan0

Build traps, all hit for real: no Ninja on this box — use Unix Makefiles + gmake, and pass -DCMAKE_C_COMPILER/-DCMAKE_CXX_COMPILER explicitly or the HIP language probe dies with CMAKE_C_COMPILER not set, after EnableLanguage; the HIP compiler is at /opt/rocm-7.2.4/lib/llvm/bin/clang, not /opt/rocm/llvm/bin/clang++; glslc is not auto-detected.

Measured performance

AMD Ryzen AI Max+ 395 / Radeon 8060S (gfx1151), 128 GB unified, GTT 131072 MiB, ROCm 7.2.4, Mesa 25.2.8 (RADV), kernel 6.17.6, Ubuntu 24.04.4, governor performance, box idle — 0 co-resident seats. -c 4096 -fa on -fit off -np 1, temperature 0, cache_prompt:false, unique nonce per request, 400 tokens, 1 warm-up discarded, median of 12.

All four tiers × both backends (MTP on, p_min 0.0)

⚠️ The prefill figures in the table above were measured on ~30-token prompts and understate real performance badly. On realistic long prompts (1,800–7,600 tokens) this model prefills at 745–813 tok/s. Short-prompt pp is not a meaningful number — quote long-prompt prefill.

What MTP is worth — STRIX_LEAN, single-shot, speculation off vs on

With enable_thinking: false, 1600 tokens

The findings that matter

1. 1.The backend crossover flips by tier. Vulkan wins decode on the 4-bit tiers; on both Q6 tiers ROCm wins outright — Q6_LEAN 64.89 vs 59.08 — and uses 7.2 GiB less GTT (28.1 vs 35.3). ROCm keeps the prefill crown everywhere (478 vs 260 on STRIX_LEAN, ~+84 %).
2. 2.MTP pays on both backends (+6.5 % / +8.9 %), acceptance 3.01–3.20 of 4 on every arm.
3. 3.Q6 buys footprint, not speed. Q6_AGENT (30.89 GiB) runs slower than STRIX_LEAN (17.88 GiB) on every workload. Choose Q6 for precision, not throughput.

⚠️ These rankings are model-specific and we say so. Measure them on your own model.

Silent-truncation gate — 480 generations, 0 truncated

On this hardware a Vulkan failure can be silent: 1 token, empty content, clean exit, /health still 200. A benchmark reporting only a median cannot see it. We count generations that actually reach n_predict:

Speculative decoding works on Vulkan on this model, across 240 Vulkan generations.

Content adequacy — clean at every budget

Non-empty content, 6 reps per cell, thinking at default:

We measured this because the architectural sibling we built alongside it — an abliterated Agents-A1 on the same qwen35moe 35B-A3B graph, same fork, same box — returns empty content on most non-code prompts at every budget up to 1600 unless enable_thinking:false is set. This model has no such problem. Same architecture, opposite behaviour: do not assume a serving parameter transfers between models on this graph.

⛔ Multi-turn agentic work — turn MTP OFF

This is the most important correction on this card. The speculative-decoding flags that make single-shot generation fast break the prompt cache on multi-turn work. The server reports:

prompt cache cold fallback: reason=spec-boundary-mismatch

Every turn then re-prefills the entire accumulated conversation. In a real agentic run we watched per-turn prefill climb to 9.5 s at 7,124 tokens while decode sat at 92.9 tok/s — the model was barely getting to generate.

Measured on an 8-turn growing conversation (each turn appends the previous answer, 300 tokens per turn, enable_thinking:false), reporting end-to-end effective throughput — total tokens generated over total wall time, which is what a user actually experiences:

Every arm with MTP enabled took 10 cache misses. Every arm without took 0. Dropping speculation costs ~19 % of single-shot decode and returns +31 % end-to-end, because prefill falls from 36 % of wall time to 9 %.

--spec-mtp-strict-qwen recovers much of the loss but does not eliminate the misses — it is not a way to keep both.

Serving config for agentic / multi-turn

llama-server -m <this-file>.gguf \
  -dev Vulkan0 -ngl 999 -fa on --no-mmap -fit off -c 131072 \
  -ctk q8_0 -ctv q8_0 -np 1 --jinja -cram 4096 \
  --host 0.0.0.0 --port 8080

Size `-cram` deliberately. It defaults to 8192 MiB per server and scales with context. A 65-token prompt costs ~63.6 MiB of cache state here, so 512 MiB holds about eight short prompts — an agentic conversation evicts itself immediately. 4096 MiB held an 8-turn session with zero misses. On a box running several models, N servers × 8 GiB of default cache will oversubscribe it.

Deployed and re-verified on the seat afterwards: 63.51 tok/s end-to-end, 0 cache misses, prefill 10 % of wall.

Use MTP only for single-shot / batch

The --spec-type draft-mtp numbers elsewhere on this card are single-shot measurements — one request, no conversation history. They are real, and they are the right choice for batch generation, summarisation, or any one-shot call. They do not describe multi-turn behaviour.

Head protection — verified, not assumed

Left to their defaults several ftypes emit a 4-bit `output.weight`, a documented quality-collapse pattern. Every file here was quantised with an explicit head flag and then read back out of the finished file by exact tensor name — never substring-matched, because attn_output matches output and gives a false pass.

On 106 token_embd is deliberately not overridden — Q5_K embeddings are the point of the LEAN tier; only the head needs forcing.

Source integrity

The source checkpoint was gated before quantisation, not assumed:

arch                 qwen35moe
tensors              753   (blk.0-39 + blk.40 MTP)
block_count          41
nextn_predict_layers 1
blk.40 tensors       20
output.weight        BF16   (unquantised source)
bytes                71,066,994,848

Quantised from BF16 only — never from an existing Q4KM or ROCmFP4.

Reproduction block

binary          llama-server  sha256 d077b98f3f04df4b482a5826c8922e39cda3aa8d2e2b2fbe1870f36d245cebe7
                llama-quantize sha256 1559189968a30781f837e4d25ca3cdf26c8eb0351023675406535eddcbe0db13
source          ROCmFPX @ d3ca53726109b46fd5cd6fa42f688c707079b44e  (-DGGML_HIP=ON -DGGML_VULKAN=ON)
hardware        AMD Ryzen AI Max+ 395 w/ Radeon 8060S (gfx1151), 128 GB unified
gtt_total       131072 MiB
rocm            7.2.4          vulkan  Mesa 25.2.8 (RADV GFX1151), API 1.4.318
kernel          6.17.6-061706-generic      os  Ubuntu 24.04.4 LTS
governor        performance
co-resident     0 seats (box idle for the whole measurement)
env             HSA_OVERRIDE_GFX_VERSION=11.5.1
serve argv      -ngl 999 -fa on -fit off -np 1 -c 4096 --jinja
                --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.0
                -dev {ROCm0|Vulkan0}
quantise argv   llama-quantize <headflag> <BF16> <OUT> <FTYPE> 16
protocol        temperature 0, cache_prompt false, unique nonce, 400 tok,
                1 warm-up discarded, median of 12

SHA256

3a4e59d1f7dfb8257ba81604f5739fa556de02f18887205f136f8fa427c8cd4f  KAT-Coder-V2.5-Dev-Abliterated-MTP-Q4_0_ROCMFP4_COHERENT.gguf
21278799c27afa5894c7626fdf6b3717b0a30fa7f8fa4b63aa831bb8e5904d01  KAT-Coder-V2.5-Dev-Abliterated-MTP-Q4_0_ROCMFP4_STRIX_LEAN.gguf
a016cd522b15666b719bc087b03f07234cc140fc663449d37f1e9dc59b4cedae  KAT-Coder-V2.5-Dev-Abliterated-MTP-Q6_0_ROCMFPX_AGENT.gguf
f92370a5b2b513b940bf46fb53c9e26898e5d85c91f84c8d85ff3bfe8bc4dfbf  KAT-Coder-V2.5-Dev-Abliterated-MTP-Q6_0_ROCMFPX_LEAN.gguf