Mojo24x7/Qwen3.6-35B-A3B-npuaware-rk3588-GGUF

Qwen3.6-35B-A3B — NPU-aware GGUF for Rockchip RK3588

A GGUF quantisation of Qwen3.6-35B-A3B built specifically for the RK3588 NPU, using the RKNPU2 backend in rk-llama.cpp.

Runs on a 16 GB Radxa ROCK 5B+: 20.4 tok/s prefill, 4.8 tok/s decode on a 1967-token prompt — the fastest prefill we have measured on this hardware, and ahead of the public quant on every axis while being 2.1 GiB smaller.

Why a special quant for this chip

The RKNPU2 backend places a weight on the NPU only if its GGUF type is one of `F16`, `Q8_0`, `Q6_K`, `Q4_0`. Everything else runs on the CPU. RK3588's matmul also requires symmetric A/B precision — there is no W4A8 — so any NPU-resident weight is read as int8 regardless of how it is stored.

So a tensor you want the NPU to accelerate (prefill, compute-bound) must be one of those four types, while a tensor you want read cheaply (decode, byte-bound) must be a non-eligible type so it stays on CPU at its real width. Prefill and decode want opposite placement, and a quant choice becomes a placement choice.

What is inside

Only the expert tensors were changed. Every attention and SSM projection is copied bit-exact from the source, so all of them remain NPU-eligible and the NPU continues to carry prefill.

Type mix: 368 × F32 · 123 × Q3K · 101 × Q80 · 83 × Q40 · 16 × Q6K · 2 × BF16.

Performance

Radxa ROCK 5B+ · RK3588 · 16 GB LPDDR5 · 4× Cortex-A76 @ 2352 MHz (pinned, performance governor) · NVMe. 1967-token prompt, 150 output tokens, prompt caching disabled, warm.

Better on every axis: prefill +33 %, decode +20 %, turn time −23 %, in a file 2.1 GiB smaller. A live 2477-token request on a cold start measured 21.5 tok/s prefill.

Choosing this model: long prompts, not long conversations

30 of the 41 blocks are Gated-DeltaNet. Recurrent state is a running summary, so it cannot be rewound to an arbitrary position the way a KV cache can. A follow-up question in the same conversation therefore re-processes most of the thread — around 94 % of the context, roughly 20 s to first token on turn two, growing with thread length.

That is inherent to the architecture rather than a property of this quantisation. It makes the model an excellent fit for long single prompts — documents, code, logs, retrieved context — and a poor fit for long back-and-forth chat, where a non-recurrent model will feel far more responsive.

Usage

Needs a rk-llama.cpp build with the RKNPU2 backend.

export RKNPU_HYBRID=W8A8_STANDARD RKNPU_GLUE=1
export LLAMA_RECURRENT_ON_CPU=1
ulimit -n 1000000

taskset -c 4-7 ./llama-server \
  -m Q35-npuaware.gguf \
  -ngl 99 --cpu-moe --no-repack \
  -np 1 -t 4 --jinja --no-warmup \
  -c 32768 -fa on -cram 2048 \
  --host 0.0.0.0 --port 8095

• —`LLAMA_RECURRENT_ON_CPU=1` is essential for this model. It assigns the recurrent layers to the CPU device, which is what allows the fused Gated-DeltaNet kernels to stay enabled. Without it both fused paths are disabled silently — no error, just a large slowdown. Check the load log for fused Gated Delta Net … enabled; it is the most important line for this architecture. (This variable is provided by the fork linked below.)
• —`RKNPU_HYBRID=W8A8_STANDARD` — selects the int8 pipeline; without it eligible weights fall to the int4 path, which is slower and worse.
• —`-fa on` here, unlike on a pure-attention model: only 10 of 41 blocks attend, so the fused kernel is roughly neutral and on edges ahead.
• —`-c 32768` is inexpensive on this architecture — KV costs about 20 KiB/token, versus roughly 96 KiB/token on a comparable dense-attention MoE.
• —`--cpu-moe --no-repack` — keeps experts memory-mapped as reclaimable page cache, which is what lets a model larger than board RAM run. --repack is on by default and will exhaust memory on a large MoE.
• —`ulimit -n 1000000` — required; each matmul context imports a DMA file descriptor and the default 1024 limit is exhausted.
• —`taskset -c 4-7 -t 4` — A76 cores only; including the A55s costs about half the throughput.

Optional: fewer active experts

Adding --override-kv qwen35moe.expert_used_count=int:4 routes 4 of 256 experts per token instead of the trained 8. On this model the trained top-8 setting costs roughly 44 % of decode speed, so the override is a large throughput win — but it is a real quality trade, so it is left out of the command above.

Reproducing

Because only the experts change, a full per-tensor type file is used so every other tensor is copied rather than requantised. llama-quantize copies a tensor when its target type already equals its current type, so listing every tensor at its current type makes each one a no-op:

# every tensor at its current type, except the three expert groups -> q3_K
python3 gen_ttypes.py \
  --targets=ffn_up_exps.weight,ffn_gate_exps.weight,ffn_down_exps.weight \
  Qwen_Qwen3.6-35B-A3B-Q4_0.gguf tt35.txt q3_K

llama-quantize --allow-requantize --tensor-type-file tt35.txt \
  Qwen_Qwen3.6-35B-A3B-Q4_0.gguf Q35-npuaware.gguf Q4_0 8

gen_ttypes.py is in the fork linked below. Worth checking the resulting tensor types: llama.cpp's mixture heuristic can promote a tensor to a higher type than requested.

Notes

• —Tuned for a 16 GB RK3588; on a 32 GB board the balance shifts.
• —This tokenizer produces roughly 6 % more tokens for the same text than Qwen3-30B (vocabulary 248,320 vs 151,936), which is worth remembering when comparing tokens/s figures between the two.
• —Deliberately not a general-purpose quant — the type choices only pay off given the NPU eligibility rules above.
• —Optimise for throughput rather than NPU utilisation. Prefill runs at low NPU occupancy while the CPU is saturated, yet moving attention off the NPU still costs a large share of prefill speed.

Thanks and credits

Sincere thanks to the Qwen team at Alibaba for releasing Qwen3.6-35B-A3B under Apache-2.0 — a remarkable hybrid architecture, and the kind of openness that makes edge work like this possible.

Thanks also to [bartowski](https://huggingface.co/bartowski), whose Qwen_Qwen3.6-35B-A3B-GGUF conversion was the starting point for this build.

• —llama.cpp — ggml-org and its contributors (MIT)
• —RKNPU2 ggml backend — invisiofficial/rk-llama.cpp, which made RK3588 NPU inference possible in the first place
• —Rockchip for the RKNN runtime, and Radxa for the ROCK 5B+

Quantisation and RK3588 tuning: Mojo24x7/rk-llama.cpp