majentik/Qwen3.6-35B-A3B-RotorQuant-MLX-2bit

[!TIP] KV-cache quantization without any fork (recommended, 2026): upstream llama.cpp/Ollama now cover this natively — use -ctk q8_0 -ctv q8_0 (~half KV memory, negligible quality loss: perplexity +0.002–0.05) or -ctk q4_0 -ctv q4_0 (~quarter memory, ≈7.6% perplexity increase). In Ollama: OLLAMA_KV_CACHE_TYPE=q8_0 with OLLAMA_FLASH_ATTENTION=1. Keep K and V types symmetric to stay on the fast fused Flash-Attention path. Since April 2026, mainline llama.cpp also applies Hadamard rotation to KV activations (PR #21038), which greatly improves low-bit KV quality (opt-out: LLAMA_ATTN_ROT_DISABLE=1). The RotorQuant/TurboQuant fork flow below is experimental/legacy: the TurboQuant llama.cpp PR was closed without merging (June 2026) and the fork is unmaintained relative to mainline. It is NOT required to use this model.

<!-- kv-upstream-note -->

Qwen3.6 35B-A3B - RotorQuant MLX 2-bit

2-bit weight-quantized MLX version of Qwen/Qwen3.6-35B-A3B with the legacy RotorQuant KV-cache fork (superseded by upstream llama.cpp KV options). Optimized for Apple Silicon inference via the MLX framework. The most aggressive quantization, fitting the full model in the smallest possible footprint. Only 3B parameters are active per token despite 35B total, making this model significantly more efficient at inference time than its parameter count suggests.

Approximate model size: ~9 GB

Model Specifications

Quickstart

import mlx.core as mx
from mlx_lm import load, generate

model, tokenizer = load("majentik/Qwen3.6-35B-A3B-RotorQuant-MLX-2bit")

prompt = "Give me a short introduction to Mixture-of-Experts models."
response = generate(model, tokenizer, prompt=prompt, max_tokens=512)
print(response)

[!IMPORTANT] Text-only extraction. This repo contains only the quantized language tower of Qwen3.6-35B-A3B. The upstream vision tower (333 tensors) and MTP head are not included, so image/video input does not work and mlx_vlm.load(...) fails with a Missing ... parameters error (the vision tower it expects is absent from the checkpoint). Load it with mlx_lm (recent version with qwen3_5_moe support) as shown above. For image/video input, use the upstream BF16 model Qwen/Qwen3.6-35B-A3B on a runtime that supports it.

About the RotorQuant / TurboQuant labels

RotorQuant and TurboQuant are this project's release labels, not distinct quantization algorithms — for any given tier, both brand repos carry byte-identical weights produced with the standard MLX / llama.cpp quantizers. No brand-specific speedup is claimed or measured. The KV-cache fork these labels originally referred to is legacy; for KV-cache memory savings use the upstream options described above (-ctk/-ctv q8_0, OLLAMA_KV_CACHE_TYPE).

KV-Cache Quantization Comparison

Memory Estimates (Qwen3.6 35B-A3B)

Hardware Requirements

This model requires approximately 9 GB of unified memory. Recommended hardware:

• —Apple M1 (16 GB+)
• —Apple M2 (16 GB+)
• —Apple M3 (16 GB+)
• —Apple M4 (16 GB+)
• —Any Apple Silicon Mac with 16 GB+ unified memory

See Also

• —Qwen/Qwen3.6-35B-A3B -- Base model
• —majentik/Qwen3.6-35B-A3B-RotorQuant-MLX-8bit -- MLX 8-bit variant
• —RotorQuant GitHub
• —MLX Framework

Quant trade-off (MLX lane)

(Current variant — 2bit — is bolded.)

Variants in this family

(Showing 24 sibling variants under majentik/qwen3.6-35b-a3b-*. The current variant — RotorQuant-MLX-2bit — is bolded.)