CoolFace
Modelpublic

sh111111111111111/Qwen3.5-9B-BitClass2-GGUF

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face
0likes258downloads
Model Card

Qwen3.5-9B — BitClass2 Mixed-Precision GGUF

Mixed-precision GGUF quantizations of Qwen3.5-9B using Hessian-informed per-tensor bit allocation. Each tensor group receives the precision level that minimizes quality loss for its measured sensitivity.

Available Quantizations

FileBPWSizePPL ↓tok/sUse Case
`Qwen3.5-9B-Q8_0.gguf`8.59.53 GB1.7286.5Near-lossless reference
`Qwen3.5-9B-Q6_K.gguf`5.66.30 GB1.8638.3High quality
`Qwen3.5-9B-Q5_K_M.gguf`5.15.70 GB1.8658.8Balanced quality and size
`Qwen3.5-9B-Q4_K_M.gguf`4.75.21 GB1.8759.2Best quality-to-size ratio
`Qwen3.5-9B-Q3_K_S.gguf`3.33.66 GB2.00111.5Maximum compression

Recommended: Q4KM — nearly matches Q6_K quality (PPL 1.875 vs 1.863) at 17% less size.

How It Compares

ModelBPWSizePPL ↓Source
ByteShape IQ3_S 3.00bpw3.03.37 GB2.069byteshape
★ Ours Q3_K_S3.33.66 GB2.001This repo
★ Ours Q4_K_M4.75.21 GB1.875This repo
★ Ours Q5_K_M5.15.70 GB1.865This repo

Our Q3KS beats ByteShape's 3.00bpw 9B on perplexity (2.001 vs 2.069 — 3.3% better), at a larger file (3.66 vs 3.37 GB). ByteShape's higher-BPW rows reach lower PPL.

The PPL curve is remarkably flat from Q6K to Q4KM: going from 6.30 GB down to 5.21 GB (saving ~1.1 GB) only costs 0.012 PPL. This is the mixed-precision allocation working — gateproj/upproj drop to Q4K while downproj and attention stay at Q6K.

Key Sensitivity Findings (Qwen3.5-9B)

The Hessian sensitivity pattern for 9B is fundamentally different from 4B:

  • —blk.3 (early layer) is most sensitive — score 1.0 for k/v. On 4B it was blk.34 (late layer).
  • —Sensitivity peaks at both ends AND middle: blk.3 (1.0), blk.7 (0.78), blk.23 (0.78), blk.27 (0.86), blk.31 (0.87)
  • —ffn_down at blk.4-5 is near-zero sensitivity (0.0003) — safe for aggressive quantization
  • —This confirms: model-specific Hessian data matters. You cannot assume late layers are always most sensitive.

How It Works

  1. 1.Hessian sensitivity — compute H_diag = mean(X²) per layer on calibration data
  2. 2.LP-optimal allocation — solve knapsack: minimize Σ(sensitivity × quant_error) subject to size ≤ target
  3. 3.Per-layer variation — within each suffix group, vary types by layer using imatrix + Hessian blend
  4. 4.GGUF export — llama.cpp --tensor-type-file for per-tensor overrides

Usage

bash
huggingface-cli download sh111111111111111/Qwen3.5-9B-BitClass2-GGUF \
    Qwen3.5-9B-Q4_K_M.gguf --local-dir .

llama-cli -m Qwen3.5-9B-Q4_K_M.gguf -cnv
llama-server -m Qwen3.5-9B-Q4_K_M.gguf --port 8080

Benchmark Details

NVIDIA GB10 ATOM (128GB unified memory, aarch64). llama.cpp commit 406f4e3. PPL via llama-perplexity (2 chunks, 851 context). tok/s via llama-bench (tg128, ngl=999).

Disclaimer

Independent project. Not affiliated with or endorsed by Qwen, Unsloth, ByteShape, Bartowski, or llama.cpp. Competitor figures are from our own benchmark harness and may differ from those projects' self-reported numbers; competitor file sizes reflect the revision we tested and may since have changed.

License

Apache 2.0, inherited from Qwen3.5-9B.