GOBA-AI-Labs/PrunedHub-Qwen3.5-35B-A3B-80pct

PrunedHub Qwen3.5-35B-A3B 80%: Expert Pruning for 24GB Mac

13.67 GB | 204/256 experts | Q3KM | MMLU -1pp, GSM8K equal | Full GPU-resident on 24GB Mac | GGUF | Apache 2.0

A pruned variant of Qwen3.5-35B-A3B that fits entirely in GPU memory on a 24GB Mac while preserving near-original quality. The official Q4KM (21.2 GB) requires swap; this model runs fully GPU-resident.

Highlights

• —13.67 GB: full GPU-resident on 24GB Apple Silicon (no swap, no SSD streaming)
• —19% smaller than original Q3KM (16.80 GB), 36% smaller than Q4KM (21.2 GB)
• —MMLU 80% (-1pp from original 81%) -- near-lossless knowledge
• —GSM8K 82% (equal to original) -- math reasoning fully preserved
• —LiveCodeBench Easy 83.1% (142 problems, contamination-free) -- strong code generation
• —DeltaNet hybrid architecture (75% Gated DeltaNet + 25% Full Attention)
• —Compatible with llama.cpp (build 8140+)

Benchmark Results

Note: LiveCodeBench Easy (142 competitive programming problems from LeetCode/AtCoder/Codeforces, 2024+) is the primary code benchmark. HumanEval 50Q is unreliable at this sample size -- LCB and HumanEval gave opposite conclusions for model comparisons in our testing.

Why Not MxMoE?

We also tested Weight-80% + MxMoE (mixed quantization, 12.69 GB) but found it damages code quality:

The extra 1 GB savings is not worth -5.6pp on code generation.

Model Details

How to Use

With llama.cpp (recommended)

Requires llama.cpp build 8140+ for Qwen3.5 DeltaNet support.

llama-server \
  -m PrunedHub-Qwen3.5-35B-A3B-80pct-Q3_K_M.gguf \
  --port 8090 \
  -ngl 99 \
  -c 4096

With OpenAI-compatible API

Once llama-server is running:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8090/v1", api_key="none")

response = client.chat.completions.create(
    model="default",
    messages=[{"role": "user", "content": "Write a Python function to find the longest palindromic substring."}],
    max_tokens=1024,
    temperature=0.7,
)
print(response.choices[0].message.content)

Requirements

• —[llama.cpp](https://github.com/ggerganov/llama.cpp) build 8140+ (for Qwen3.5 / DeltaNet support)
• —24 GB RAM for full GPU-resident inference (model uses ~13.7 GB; rest for KV cache)
• —Apple Silicon (Metal) recommended; CUDA also supported

Compression Methodology

Weight-Based Expert Pruning (80% Keep)

Each expert is scored using weight magnitude importance -- the L2 norm of the expert's FFN weight matrices:

importance(layer, expert) = ||gate_weight||_F + ||up_weight||_F + ||down_weight||_F

The bottom 20% of experts per layer are removed (256 -> 204 experts/layer). Weight-based scoring outperformed activation-based calibration for this model:

Why Weight-Based Wins for 256 Experts

With 256 experts per layer, activation-based calibration (55 prompts x 32 tokens) cannot adequately cover all experts. Weight magnitude, which encodes information from the full training data, provides a more reliable importance signal. This is consistent with findings that weight-based pruning dominates for high expert-count models.

Size Breakdown

Original Q4_K_M:      21.20 GB (256 experts x 40 layers)
Original Q3_K_M:      16.80 GB (256 experts x 40 layers)
After 80% pruning:    13.67 GB (204 experts x 40 layers, Q3_K_M)
Total reduction:      -18.6% (from Q3_K_M), -35.5% (from Q4_K_M)

Pruning Curve (Qwen3.5-35B-A3B)

Qwen3.5-35B-A3B vs Qwen3.5-27B (Dense)

The Dense 27B model outperforms the MoE 35B-A3B on many benchmarks, but at 9x inference cost:

MoE's value is cost efficiency: 3B active parameters achieve 85-93% of the Dense 27B's quality. And only MoE models can be further compressed via expert pruning.

Limitations

• —Japanese quality: -15pp (100% -> 85%) due to pruning of some language-specialized experts. Expert Tuning could recover this
• —Code generation: -9.9pp on LiveCodeBench Easy. Trade-off for 19% size reduction
• —No post-pruning training: Pruned without fine-tuning. Quality could be improved with expert tuning
• —llama.cpp 8140+ required: Older builds do not support Qwen3.5 / DeltaNet architecture

Related Models

Citation

@misc{goba2026qwen35prune,
  title   = {Weight-Based Expert Pruning for Qwen3.5-35B-A3B},
  author  = {GOBA-AI-Labs},
  year    = {2026},
  url     = {https://huggingface.co/GOBA-AI-Labs/PrunedHub-Qwen3.5-35B-A3B-80pct},
  note    = {80\% expert retention achieves 19\% size reduction with MMLU -1pp on 256-expert DeltaNet MoE}
}

Acknowledgments

• —Qwen Team for releasing Qwen3.5-35B-A3B under Apache 2.0
• —Unsloth for the Q3KM quantization with imatrix
• —llama.cpp for DeltaNet support and GGUF format

License

This model inherits the Apache 2.0 License from the base model (Qwen/Qwen3.5-35B-A3B).