varjosoft/Qwen3.6-35B-A3B-TQ-apex3

Qwen3.6-35B-A3B-TQ-apex3

Data-driven mixed-precision native TurboQuant checkpoint of `Qwen/Qwen3.6-35B-A3B`. Extends `-TQ-apex2` by skipping the shared-expert down-projection — the tensor family our per-tensor kurtosis scan flagged as 100 % FP16-bucket (κ mean 16.3, max 67, 41 of 41 tensors above threshold). apex2 raised those to TQ4; apex3 leaves them uncompressed.

This is the endpoint of PAT-0349 Kurtosis-Aware Automatic Bit-Width Selection for this model: every tensor family whose measured κ puts it in the FP16 bucket is now actually stored as FP16.

What changed from apex2

Two deliberate bit-flips from apex2:

1. 1.`shared_expert.down_proj` → FP16 (41 tensors, ~600 MB). The largest remaining κ-bucket mismatch in apex2 — TQ4's 16 centroids can't represent a tensor whose distribution has 67× Gaussian tail mass without discarding outliers. Leaving them uncompressed is cheap on disk (~600 MB on 18 GB) and eliminates the main residual quality leak.
2. 2.Routed `down_proj` → TQ3 (41 tensors, −40 MB). apex / apex2 caught these under the generic down_proj substring in _SENSITIVE_PATTERNS and bumped them to TQ4 — but routed down_proj is κ mean 3.4, which PAT-0349 puts in the TQ3 bucket. TQ4 was wasted bits.

Net disk-size delta: apex2 → apex3 is +600 MB − 40 MB ≈ +560 MB. Expected total ~18.6 GB.

Measured quality and throughput

Benched on M4 Pro 48 GB, gsm8k-200 5-shot CoT @ max_tokens=1024:

apex3 is the quality-leading TurboQuant checkpoint for this model: +2.0 ppt over the MLX-4bit reference while being 1 GB smaller on disk. The sharedexpert.downproj→FP16 upgrade over apex2 adds +0.5 ppt — a small but measurable effect of honoring the 100 % FP16-bucket flag from the κ profile.

How to use

Same loader as -TQ-apex2:

pip install git+https://github.com/varjoranta/turboquant-vllm.git@feat/mixed-bits-mlx-loader
huggingface-cli download varjosoft/Qwen3.6-35B-A3B-TQ-apex3 \
    --local-dir ~/models/qwen3.6-35b-a3b-tq-apex3

The fp16-kept tensors pass through mlx_lm's standard weight-load path; TQ3/TQ4 tensors carry the TurboQuant .tq_packed sidecars and dispatch into the tq3_gemv_bs1_mlx / tq4_gemv_bs1_mlx Metal kernels automatically.

Compression recipe (tensor-family)

• —Skip (fp16): lm_head, embed, norm, bias, shared_expert.down_proj, .gate.weight (router), in_proj_b, out_proj, linear_fc (GatedDeltaNet)
• —TQ4 (sensitive): o_proj, q_proj, k_proj, v_proj, shared_expert.gate_proj, shared_expert.up_proj — caught by "shared_expert" pattern (after shared_expert.down_proj is already skipped)
• —TQ3 (default): routed expert gate_proj / up_proj / down_proj (bulk of params)
• —Group size: 128; rotation: Walsh-Hadamard + seed=42 random signs; codebook: Lloyd-Max scalar

Reproduce

git clone https://github.com/varjoranta/turboquant-vllm
cd turboquant-vllm
uv venv --python 3.12 && uv pip install -e . accelerate "transformers>=5.5" torch

python3 - <<'PY'
import turboquant_vllm.weight_quant as wq
import turboquant_vllm.checkpoint as cp

# NOTE: no generic "down_proj" — routed down_proj is κ=3.4 and belongs in TQ3.
wq._SENSITIVE_PATTERNS = (
    "o_proj", "q_proj", "k_proj", "v_proj",
    "shared_expert",   # gate_proj + up_proj → TQ4 (down_proj caught by SKIP first)
)
cp._SENSITIVE_PATTERNS = wq._SENSITIVE_PATTERNS

wq._SKIP_PATTERNS = tuple(list(wq._SKIP_PATTERNS) + [
    "shared_expert.down_proj",  # κ mean 16.3, max 67
    ".gate.weight",              # MoE router — κ mean 17.5
    "in_proj_b",                 # GatedDeltaNet
    "out_proj",                  # GatedDeltaNet
    "linear_fc",                 # GatedDeltaNet
])
cp._SKIP_PATTERNS = wq._SKIP_PATTERNS

cp.save_tq3_checkpoint(
    model_id="Qwen/Qwen3.6-35B-A3B",
    output_dir="./qwen3.6-apex3",
    bits=3, sensitive_bits=4, group_size=128,
)
PY

Needs ≥ 100 GB CPU RAM (full bf16 load during compression). Inference: ~20 GB resident.

Citations

@article{qwen2026qwen36,
  title={Qwen3.6-35B-A3B}, author={Qwen Team, Alibaba}, year={2026},
  url={https://huggingface.co/Qwen/Qwen3.6-35B-A3B}
}
@inproceedings{malinovskii2025higgs,
  title={HIGGS: Pushing the Limits of Large Language Model Quantization via
         Hadamard Rotations and MSE-Optimal Grids},
  author={Malinovskii, Vladimir and Mazur, Andrei and Ilin, Ivan and Kuznedelev,
          Denis and Burlachenko, Konstantin and Yi, Kai and Alistarh, Dan and
          Richtarik, Peter},
  booktitle={NAACL}, year={2025},
  url={https://aclanthology.org/2025.naacl-long.543/}
}

Mixed-precision bit-width selection draws on PAT-0349 (Kurtosis-Aware Automatic Bit-Width Selection) and the MXPLM survey (Mixed-Precision Quantization for Language Models, arXiv, Oct 2025).

License

Inherits Apache-2.0 from the base model.

Links

• —Compression / loader: `varjoranta/turboquant-vllm` (feat/mixed-bits-mlx-loader)
• —Predecessors: `-TQ-apex2`, `-TQ-apex`, `-TQ3-native`