sh111111111111111/Qwen3.5-35B-A3B-BitClass3-GGUF

Qwen3.5-35B-A3B — BitClass3 Mixed-Precision GGUF

Mixed-precision GGUF quantizations of Qwen3.5-35B-A3B, a 256-expert (8 routed + 1 shared active) Mixture-of-Experts model with hybrid Gated-DeltaNet + attention layers (~3B active parameters per token). Each file is produced with llama.cpp's --target-bpw per-tensor allocator (PR #15550), which assigns a quantization type to every tensor — attention, the 256-way expert FFN, shared experts, and the DeltaNet/SSM tensors — to hit a target bits-per-weight with minimum imatrix-weighted error.

These are text-only GGUFs of the language model (the vision tower is not included) with the NextN / multi-token-prediction head removed — standard text-generation quants for use with llama.cpp.

Available Quantizations

Perplexity (PPL) is measured on wikitext-2 (-c 2048, full file); the BF16 source scores 5.868. KL-divergence (KLD) against the BF16 source is the primary fidelity metric, reported as the mean and the robust 99.9th percentile.

Recommended: Q4KM sits within ~0.74% of BF16 perplexity at 20.97 GB (KL-mean 0.020). Q5KM is within ~0.2% of BF16 PPL at 24.50 GB. From Q5 upward the levels sit within PPL measurement noise (±0.04) of each other and of BF16, so they can be chosen by size; KL-divergence keeps falling across that range (0.017 → 0.008 → 0.005) and is the finer discriminator at the high-bpw end.

Fidelity to the BF16 source

Quality is reported against the model's own BF16 source. The file names (Q4_K_M, Q6_K, …) denote the target tier each mix is built around; the per-tensor allocation is solved to a BPW target and can be inspected with gguf-dump. A per-tensor type manifest is included in this repo as manifest.md.

Task Accuracy (quantization fidelity)

HellaSwag (2000 tasks) and Winogrande (1267 tasks), scored with llama.cpp log-likelihood on the BF16 source and on each quant under one identical harness. The signal of interest is the delta vs BF16, which isolates how faithfully each quant preserves the source model's behavior. These are log-likelihood fidelity measurements, not the model's headline generative capability.

From Q4KM upward, both tasks sit within measurement noise of BF16 (≈±1.7 HellaSwag / ±2.4 Winogrande, 95% CI at these task counts). Q3KS shows a HellaSwag delta of −1.75, near the ±1.7 threshold. This tracks the PPL/KLD picture: above ~Q4 the quants preserve the source model's task behavior.

Generative reasoning (indicative)

A reproduction of two public benchmarks, run generatively (served with llama-server, thinking on, the answer parsed from the response) on BF16 and two quants. Every question was scored on a full thinking pass (up to a 32K-token budget at 40K context); all questions produced a parsed answer.

n = 80 per benchmark (95% CI ≈ ±8–9 points), so these are indicative. The reliable signal is quant-vs-BF16. Reference figures are Qwen's official Qwen3.5-35B-A3B scores, shown for context (these differ from the newer Qwen3.6-35B-A3B at 85.2 / 86.0, which is a separate model).

Across both tasks the quants fall within the ±8–9 point sampling noise of BF16; at n = 80 this test resolves differences of roughly ±8 points, so it complements (rather than replaces) the much finer PPL/KLD fidelity measurements above.

How It Works

1. 1.Per-tensor allocation — llama.cpp's --target-bpw solver (PR #15550) assigns a quantization type to every tensor directly, hitting the target BPW with minimum imatrix-weighted error. This scales to the MoE's tens of thousands of expert tensors without a hand-written per-layer recipe.
2. 2.imatrix — an importance matrix computed over wikitext guides the per-tensor error weighting.
3. 3.NextN/MTP head removed — the source ships a multi-token-prediction (NextN) head as a trailing block; it is stripped so these are standard text-generation quants (block_count 41 → 40).
4. 4.Text-only — the vision tower / mmproj is not included; these GGUFs contain the language model only. This, with the stripped MTP head, accounts for the parameter count being below the upstream repo's listed total.

Usage

hf download sh111111111111111/Qwen3.5-35B-A3B-BitClass3-GGUF \
    Qwen3.5-35B-A3B-Q4_K_M.gguf --local-dir .

# -c sets context length; raise it toward the model's 262K native limit as RAM allows.
# Sampling below uses Qwen's recommended thinking-mode settings; the presence penalty
# matters for this model.
llama-cli    -m Qwen3.5-35B-A3B-Q4_K_M.gguf -cnv -c 20480 \
    --temp 1.0 --top-p 0.95 --top-k 20 --presence-penalty 1.5
llama-server -m Qwen3.5-35B-A3B-Q4_K_M.gguf --port 8080 -c 20480 --jinja \
    --temp 1.0 --top-p 0.95 --top-k 20 --presence-penalty 1.5

Per Qwen's card, thinking-mode general tasks use temperature=1.0, top_p=0.95, top_k=20, presence_penalty=1.5 (use presence_penalty=0.0 for code). Requires a current llama.cpp build with qwen3_5_moe architecture support.

Benchmark Details

NVIDIA GB10 (128 GB unified memory, aarch64). llama.cpp with --target-bpw (PR #15550). The imatrix is computed from wikitext; wikitext-2 PPL via llama-perplexity (-c 2048, full file); KLD via llama-perplexity --kl-divergence against the BF16-source logits (mean / median / 99.9th percentile reported). The BF16 reference is the converted, MTP-stripped GGUF of the language model. Task accuracy via llama-perplexity --hellaswag (2000 tasks) and --winogrande (1267 tasks). Generative MMLU-Pro / GPQA-Diamond via llama-server (thinking on) at n = 80 each, as small-sample reproductions.

Disclaimer

Independent project. Not affiliated with or endorsed by Qwen or the llama.cpp project. All benchmark numbers here are from our own measurements on the hardware and harness described above; Qwen's official figures, where shown, are labelled as such and are provided for context.

License

Apache 2.0, inherited from Qwen3.5-35B-A3B.