Mojo24x7/Qwen3.5-9B-HighIQ-rkllm-rk3588

Qwen3.5-9B-Claude-4.6-HighIQ-INSTRUCT — RKLLM w8a8_g512 for RK3588

DavidAU/Qwen3.5-9B-Claude-4.6-HighIQ-INSTRUCT converted to Rockchip RKLLM format for the RK3588 NPU. Built and verified on a Radxa ROCK 5B+ (16 GB).

Requirements

• —RK3588 board with ≥ 16 GB RAM — the model is 9.5 GiB resident and leaves little headroom; it will not fit alongside another large model
• —rknpu driver 0.9.8+ (sudo cat /sys/kernel/debug/rknpu/version)
• —RKLLM runtime 1.3.0 from airockchip/rknn-llm
• —`max_context_len` at runtime must be ≤ the baked 16384. Practical values are lower: on a 16 GB board 4096–8192 is realistic once the 9.5 GiB of weights plus KV are accounted for.

Measured

Radxa ROCK 5B+ (RK3588, 16 GB, performance governors on CPU/DMC/NPU, 3 NPU cores):

• —~2–4 tokens/s generation. This is a 9B on a 6 TOPS NPU — usable for chat and background/agent work, not for anything latency-sensitive.
• —Prompt length dominates first-token latency: the same short answer took 18 s at 12 prompt tokens and 43 s at 820 prompt tokens.
• —Coherent, well-structured English. Long-form generation completes and stops naturally — no ignore_eos workaround needed (see below, this is not true of every grouped-quant build in circulation).

★ Two conversion findings that decide whether this works at all

Both were arrived at the hard way, and either one gets you a model that loads fine and produces nonsense.

1. Grouped quantization is required at 9B. Per-tensor `w8a8` is not usable.

A per-tensor w8a8 build of this model loaded and generated, but the output was incoherent — rambling, self-correcting loops, drifting language, wrong content for the prompt, while still answering trivial factual questions correctly (which is exactly what makes it easy to miss). RK3588 supports w8a8, w8a8_g128, w8a8_g256, w8a8_g512; per-group scales preserve far more fidelity than one scale per tensor. w8a8_g512 was sufficient here. Note this is a heavily merged/abliterated 9B, which is inherently harder to quantize cleanly than a plain base model.

2. Calibration data must match the target language.

The calibration file shipped with the RKLLM examples is Chinese. Using it produced a model that drifted into Chinese and Polish, ignored English instructions, and generated off-topic content — while still answering "capital of France" correctly in English. The fix was a 72-sample English instruction/coding calibration set. Verify your calibration data contains no CJK and, when you test, check the output language, not merely that tokens appear.

A related trap: calibrating a text model using the multimodal export path's image-text samples produces degenerate, repetitive output.

Note on grouped-quant builds and spurious early EOS

Some community grouped-quant RKLLM builds (e.g. a Qwen3-8B w8a8_g512 we tested) truncate mid-word after ~10 tokens. That is not a fidelity problem and not a property of grouped quantization: the model has an abnormally high EOS logit and emits a spurious end-of-sequence token early. Setting ignore_eos_token = true makes such a model produce the full, correct answer — proving the weights were fine all along.

This build does not need that. Its EOS behaviour is healthy and it stops naturally. Mentioned only because if you hit early truncation on some other .rkllm, the cause is usually EOS handling rather than the quantization.

Credits

• —[DavidAU/Qwen3.5-9B-Claude-4.6-HighIQ-INSTRUCT](https://huggingface.co/DavidAU/Qwen3.5-9B-Claude-4.6-HighIQ-INSTRUCT) — the base model (Apache-2.0). All capability and all training/merging work is DavidAU's; the model's name and its provenance claims are theirs, not ours. This repo only changes the numeric format.
• —Qwen team — the underlying Qwen3.5 architecture.
• —[airockchip/rknn-llm](https://github.com/airockchip/rknn-llm) — RKLLM toolkit and runtime.

License

Apache-2.0, inherited from the base model. Format conversion only — no retraining, no merging, no modification beyond quantization.