CoolFace
Modelpublic

inductiveML/Qwen3.6-35B-A3B-evolved-mxbit

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face
2likes212downloads
Model Card

Qwen3.6-35B-A3B — Evolved Mixed-Bit (MLX, 12.63 GB)

TL;DR: 12.63 GB MLX quant of Qwen3.6-35B-A3B. 16.8% smaller than the public TurboQuant-3bit MLX quant at statistically tied WikiText-2 perplexity. Loads with stock mlx-lm, no custom loader.

A per-module mixed-precision quantization of `Qwen/Qwen3.6-35B-A3B` for Apple Silicon / MLX. The bit-width and group-size of every quantized tensor were chosen by an evolutionary search (OpenEvolve / MAP-Elites) that minimizes memory under a perplexity budget — not a uniform setting.

Why this exists

Apple Silicon local-LLM users usually get a bad tradeoff:

  • —4-bit works, but costs memory
  • —3-bit saves memory, but quality can fall off
  • —clever custom quant recipes often need custom loaders

This release is the boring-useful point: smaller than the known 3-bit quant, same measured perplexity, and still just `mlx_lm.load(...)`.

Results

Perplexity on WikiText-2-raw test (~16,260 tokens, identical eval pipeline for every model, including the real public TurboQuant model). Lower is better.

ModelSizePerplexity ↓vs TurboQuant-3bit
Qwen3.6-35B-A3B-4bit (base)19.51 GB4.8308—
Qwen3.6-35B-A3B-TurboQuant-MLX-3bit15.18 GB5.6743—
this model (evolved)12.63 GB5.6673−16.8 % size, −0.007 ppl (tied)

≈ 2.6 effective bits/weight. The Δppl vs TurboQuant (−0.007) is within measurement noise, so read this as "smaller at equal quality," not a quality win.

Usage

bash
uv tool install mlx-lm
mlx_lm.generate --model inductiveML/Qwen3.6-35B-A3B-evolved-mxbit \
  --prompt "Write a short Python function to parse a CSV file."

Or from Python:

python
from mlx_lm import load, generate
model, tokenizer = load("inductiveML/Qwen3.6-35B-A3B-evolved-mxbit")
print(generate(model, tokenizer, prompt="The capital of France is", max_tokens=64))

Quantization recipe

The fused routed experts (mlp.switch_mlp.*) carry ~90% of the weight bytes, so the search spends its budget there and protects everything small and sensitive:

ComponentPrecision
Routed experts — down_proj (all layers)3-bit
Routed experts — gate/up_proj, layers < 93-bit
Routed experts — gate/up_proj, layers ≥ 92-bit
MoE routers (mlp.gate, shared_expert_gate)8-bit (base, unchanged)
Embeddings, lm_head, attention, shared expert4-bit (base, unchanged)

Deeper experts also use group_size=128 (vs 64) to shave overhead. The exact per-module policy is in the config.json quantization field (512 entries).

How it was made

The quantization "recipe" is a function choose_bits(path, info) → {bits, group_size} evolved with OpenEvolve (MAP-Elites quality-diversity search). Objective: minimize total bytes subject to perplexity degradation ≤ the public TurboQuant-3bit level. Candidates are scored on real measured bytes and perplexity, and the frontier was re-validated on a held-out test corpus the search never saw.

Evaluation & honesty

  • —This is not a general "better model" claim. It is a memory/quality tradeoff claim, measured on the same WikiText-2 eval pipeline against the public TurboQuant model.
  • —All perplexities above were measured with the same code on the same ~16k-token WikiText-2 test split, including the real TurboQuant model — apples-to-apples, not cross-paper numbers.
  • —The size win (16.8%) is exact (real stored bytes). The quality difference (−0.007 ppl) is within noise → "smaller at equal quality."

License

Apache-2.0, inherited from the base model Qwen/Qwen3.6-35B-A3B.