cafonez/Escha-W2-35B-A3B-ROCmFP2

Qwen3.6-35B-A3B-Escha-W2 → ROCmFP2 (GGUF)

A GGUF transcode of EschaLabs/Qwen3.6-35B-A3B-Escha-W2 into the ROCmFPX quantization formats, for AMD Strix Halo / gfx1151 and other ROCm/Vulkan targets.

Escha's original export is a proprietary eschamoe packed format that only ships a CUDA decoder. This repo contains the same expert weights decoded and repacked into Q2_0_ROCMFPX, so the model runs on AMD hardware through the ROCmFPX llama.cpp fork.

This file will not load in upstream llama.cpp, Ollama, LM Studio, koboldcpp or llama-cpp-python. It uses ggml tensor type IDs 107 (Q2_0_ROCMFPX) and 103 (Q8_0_ROCMFPX), which are outside upstream's range (GGML_TYPE_COUNT = 43). You need the fork below. The qwen35moe architecture itself is upstream — only the quantization types are fork-specific.

Requirements

Build **charlie12345/ROCmFPX** at commit 41db2f9eb or later:

git clone https://github.com/charlie12345/ROCmFPX
cd ROCmFPX
cmake -B build -DGGML_VULKAN=ON -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151
cmake --build build -j

The model itself loads on any build from db6844d9b (build 172) onward — the quantization types, the qwen35moe architecture and draft-mtp are all present there. Only --spec-mtp-strict-qwen requires 41db2f9eb+.

Pin the commit if you can. The ROCmFPX type IDs live in a private-use range and could renumber between fork revisions, which would silently mismatch an older build.

Contents

About the MTP head

Escha's export ships the nextn (MTP) block without its routed experts, so a GGUF built from it alone fails to load with missing tensor 'blk.40.ffn_down_exps.weight'. Those two tensors were grafted from the base Qwen/Qwen3.6-35B-A3B at Q4_0.

They are not Escha weights. This only affects the draft model — every speculative token is verified against the target model before being accepted, so drafting from base-model weights cannot introduce wrong tokens — and with --spec-mtp-strict-qwen the output is bit-identical to non-speculative greedy decoding. The graft costs 453 MB and buys a ~1.5× decode speedup.

If you don't want it, strip block 40 and set qwen35moe.block_count=40, qwen35moe.nextn_predict_layers=0 — that yields a 12.65 GB file (+2.6% over Escha's original, versus +6.3% with the head).

Recommended flags

Measured on Strix Halo (Radeon 8060S, gfx1151), Vulkan backend.

llama-server \
  -m Qwen3.6-35B-A3B-Escha-W2-ROCmFP2.gguf \
  -np 1 \
  -dev Vulkan0 --spec-draft-device Vulkan0 \
  -ngl 999 --spec-draft-ngl all \
  -fa on --no-mmap \
  -ctk f16 -ctv f16 \
  -c 131072 -b 2048 -ub 512 \
  --jinja \
  --spec-type draft-mtp \
  --spec-draft-n-max 4 \
  --spec-draft-n-min 0 \
  --spec-draft-p-min 0.15 \
  --spec-mtp-strict-qwen

`--spec-mtp-strict-qwen` needs `41db2f9eb` or newer (added in PR #54). Older binaries reject the flag — if yours does, either update or drop that line. Everything else works without it and the speed figures are unaffected, since strict costs only ~0.5%. What you lose is the guarantee that speculative output is bit-identical to non-speculative greedy decoding; drafts are verified against the target model either way, so output remains valid.

--spec-type is a llama-server flag; llama-completion rejects it. Note also that with --jinja, the raw /completion endpoint returns a single token — benchmark through /v1/chat/completions.

Speculative decoding tuning

--spec-mtp-strict-qwen requires -np 1; the server refuses to start otherwise.

Two things worth internalizing:

1. 1.`p_min` matters far more than `n_max`. The default p_min=0.75 leaves ~20 t/s on the table. Drafting aggressively (lower p_min) wins even though per-draft acceptance drops.
2. 2.Long drafts are actively harmful on this MoE. n_max=8 falls below the no-speculation baseline despite a higher mean acceptance length (4.12), because each extra draft position widens the union of experts that must be gathered. Per-position acceptance is (0.982, 0.600, 0.327, 0.145) — the 4th token is rarely worth it.

Strict verification is nearly free (95.5 vs 96.1 t/s, ~0.5%) and guarantees output identical to non-speculative greedy decoding. Leave it on.

Reproducibility

A clean build of main was benchmarked against the development build on the same model and flags, and they are indistinguishable — pp512 1252 vs 1256 t/s, tg128 65.11 vs 65.09 t/s, well inside run-to-run noise. The public repo delivers the full ROCmFP2 performance; nothing here depends on unpublished work. See the variance note below before drawing conclusions from small differences.

Throughput

Use Vulkan for decode — it beats ROCm by 36% (63.1 vs 46.4 t/s). ROCm is marginally ahead on prompt processing, within noise. Flash-attn on/off and ubatch 1024 changed nothing measurable at short context.

Long context

Decode degrades roughly linearly with depth, since ~10 of 40 layers are full attention (full_attention_interval: 4) while the rest are linear-attention:

KV cache at 256k with f16 is ~6 GB; total resident ~21 GB.

Benchmarking caveat. Run-to-run variance on this platform is ±7 t/s through the chat API — identical configs produced 81, 85 and 96 t/s on consecutive runs. Warm up and average at least 3 runs before trusting any comparison. Differences between f16 and q4_0 KV, and between 32k/64k/131k context, were within noise in our testing.

Quality

Verified working: coherent prose, chain-of-thought reasoning, Python code generation, and structured tool calling (correct tool_calls with valid JSON arguments and finish_reason: tool_calls).

Reconstruction fidelity was validated against the original Qwen/Qwen3.6-35B-A3B BF16 weights before quantization — decoded experts correlate 0.953 (gateup, 2-bit source) and **0.990** (downproj, 3-bit source) with the base model.

No standardized benchmark scores (MMLU, BFCL, etc.) have been run on this transcode yet. Escha reports boolq 88.38 vs 88.04 for their original quantization; expect some additional loss here, since repacking to 2.5 bpw adds error on top of Escha's own 2-bit.

Provenance

EschaLabs/Qwen3.6-35B-A3B-Escha-W2 is a quantization of Qwen/Qwen3.6-35B-A3B. This is verifiable rather than assumed: layer-0 expert-0's gate_up has 560 of 1024 all-zero rows in the base model, and Escha's escha_rout has zeros at exactly those same 560 positions.

The transcode decodes Escha's packed codes, applies the documented reconstruction chain W_eff = diag(rin) · H · W_bare · H · diag(rout) (where H is the normalized 128-wide block Hadamard), then repacks to ROCmFPX. Escha's s_in/s_out scales are all-ones in this export (already folded).

License and attribution

Apache-2.0, inherited from EschaLabs' release ("model weights only, released under the Apache License, Version 2.0"). Credit to:

• —EschaLabs — the W2 quantization this is derived from
• —Qwen — the Qwen3.6-35B-A3B base model
• —charlie12345/ROCmFPX — the fork and quantization formats
• —exllamav3 / turboderp — the trellis codebook design the Escha format builds on