sh111111111111111/Qwen3.5-9B-BitClass-GGUF
Qwen3.5-9B — BitClass Mixed-Precision GGUF
Mixed-precision GGUF quantizations of Qwen3.5-9B using learned per-tensor quantization profiles. Each tensor group receives the precision level that minimizes quality loss for its importance — more bits where they matter, fewer where they don't.
Qwen3.5-9B uses a hybrid DeltaNet + Attention architecture (24 DeltaNet layers + 8 standard attention layers). Our pipeline includes full support for DeltaNet's unique tensor groups (inprojqkv, inprojz, out_proj) alongside standard attention and MLP tensors — 10 suffix groups, 250 weight tensors total.
Seven precision levels from compact (3.0 bpw) to high quality (5.0 bpw).
Models
Target BPW is the planner's per-tensor bit budget (and the filename label). The actual whole-file BPW runs ~0.3–0.4 higher, because output/embedding tensors are kept at higher precision and GGUF carries metadata overhead — see the Size column for the real footprint.
Recommended: MX-3.5bpw for the best quality-to-size ratio. MX-3.0bpw for maximum compression. MX-4.5bpw for the highest quality in this ladder.
How It Compares
All models benchmarked in the same session on identical hardware (NVIDIA GB10 ATOM, GPU) for fair comparison.
Key Results
- Near-parity with ByteShape at low BPW: MX-3.0 (PPL 2.099) vs ByteShape 3.00 (PPL 2.069) — 1.4% behind, at a larger file (3.79 vs 3.37 GB)
- Competitive at mid-range: MX-3.5 (PPL 1.957) vs ByteShape 3.60 (PPL 1.947) — 0.5% behind, at 4.38 vs 4.04 GB
- Beats Bartowski Q3_K_S: MX-4.5 (PPL 1.879) vs Bartowski Q3KS (PPL 1.898) — 1.0% better, at a larger file (5.24 vs 4.93 GB)
DeltaNet Architecture
Qwen3.5-9B is not a standard transformer. It uses a hybrid architecture:
- 24 DeltaNet layers with linear attention (inprojqkv, inprojz, out_proj tensors)
- 8 standard attention layers (qproj, kproj, vproj, oproj)
- 32 MLP layers (gateproj, upproj, down_proj)
Our pipeline handles all 10 tensor suffix groups with appropriate quantization profiles for each.
Running with llama.cpp
# Chat
llama-cli -m Qwen3.5-9B-MX-3.5bpw.gguf -cnv
# Server (OpenAI-compatible API)
llama-server -m Qwen3.5-9B-MX-3.5bpw.gguf --port 8080
# Benchmark
llama-perplexity -m Qwen3.5-9B-MX-3.5bpw.gguf -f your_eval_data.txtBenchmarking Details
All benchmarks run with llama.cpp (commit 406f4e3) on NVIDIA GB10 ATOM GPU with full offload (-ngl 999). Perplexity measured via llama-perplexity on a held-out evaluation set (20 chunks, 512 context). Throughput via llama-bench (512 prompt / 128 generation tokens). All models benchmarked in the same session.
Disclaimer
Independent project. Not affiliated with or endorsed by Qwen, Unsloth, ByteShape, Bartowski, or llama.cpp. Competitor figures are from our own benchmark harness and may differ from those projects' self-reported numbers; competitor file sizes reflect the revision we tested and may since have changed.
License
Apache 2.0, inherited from Qwen3.5-9B.