cahlen/qwen3.5-35b-a3b-compacted-GGUF

qwen3.5-35b-a3b-compacted-GGUF

GGUF quantizations of `continuum-ai/qwen3.5-35b-a3b-compacted` — a compacted MoE model pruned from Qwen3.5-35B-A3B (89 experts removed, 30% smaller) while preserving reasoning quality.

All low-bit quants (Q3, Q2, IQ) are calibrated with an importance matrix* for best quality at each size.

Available Quantizations

How to Use

With llama.cpp

llama-cli -m qwen3.5-35b-a3b-compacted-Q4_K_M.gguf -p "Hello" -ngl 999

With llama.cpp server

llama-server -m qwen3.5-35b-a3b-compacted-Q4_K_M.gguf -c 4096 -ngl 999

With Ollama

ollama run hf.co/cahlen/qwen3.5-35b-a3b-compacted-GGUF:Q4_K_M

With LM Studio

Download any GGUF file above and load it in LM Studio.

Choosing a Quant

About the Source Model

This is a compacted version of Jackrong/Qwen3.5-35B-A3B-Claude-4.6-Opus-Reasoning-Distilled created by continuum-ai using Plasticity Compaction — a technique that prunes underutilized MoE experts based on runtime activation profiling:

• —256 experts reduced to 167 (-35%)
• —67GB reduced to 47GB BF16 (-30%)
• —Chain-of-thought reasoning and code generation quality preserved

Perplexity Evaluation (WikiText-2)

Lower is better. BF16 is the unquantized baseline.

Key takeaways:

• —Q8_0 through Q4_K_M: Negligible quality loss (<2.2%) — safe for all use cases
• —Q3_K_M / IQ3_M: Moderate degradation (~6%) — good for constrained hardware
• —Q2_K / IQ2_M: Noticeable degradation (~20%) — usable for casual use
• —IQ1_M: Significant quality loss — only for extreme VRAM constraints

Quantization Details

• —Quantized by: cahlen
• —Importance matrix: Generated from WikiText-2 (200 chunks) on NVIDIA RTX 5090
• —Tool: llama.cpp
• —Hardware: NVIDIA RTX 5090 32GB / Intel Core Ultra 9 285K / 188GB RAM