Modelpublic

stamsam/Qwen3.6-28B-REAP-mixedbit-tuned-mlx

sourceHugging Faceupdated 4mo agoView on Hugging Face

2likes55downloads

Model Card

Qwen3.6-28B-REAP-mixedbit-tuned-mlx

An MLX "mixedbit" quantization of `0xSero/Qwen3.6-28B` (a REAP-pruned Qwen3.5/3.6 hybrid MoE), shipped with the [buun hardened chat template](https://huggingface.co/spiritbuun/buun-Qwen3.6-chat_template) and a documented, tested inference config.

The weights are not fine-tuned. This repo's value is: (1) the mixed-bit MLX quant (~13 GB), (2) the bundled hardened chat template, and (3) a battle-tested deployment config — see below.

What is "mixedbit"?

A non-uniform MLX quantization: different layers are stored at different bit-widths instead of a single uniform 4-bit. The result here is ~13 GB on disk — actually smaller than a standard uniform 4-bit quant of the same model (~16–17 GB), because some layers are pushed below 4 bits while sensitive layers are kept higher. It runs well on Apple Silicon via mlx-lm / oMLX.

Architecture: qwen3_5_moe — a hybrid linear-attention + MoE model (40 layers, fused 3-D expert tensors, ~3B active params per token). This matters for tooling (see "Known incompatibilities").

Lineage & credits

—Base / weights: `0xSero/Qwen3.6-28B` — REAP 20%-pruned from Qwen/Qwen3.6-35B-A3B, LoRA-SFT'd on Claude Opus reasoning traces. All model capability is theirs. This repo only re-quantizes those weights to MLX mixedbit.
—Chat template: `spiritbuun/buun-Qwen3.6-chat_template` — hardened Qwen3.5/3.6 template (fixes the official template's "empty-think poisoning" bug). The original stock template is preserved here as chat_template.STOCK.jinja.bak.
—Base architecture: Qwen3.5/3.6 by Alibaba/Qwen.

Please honor the licenses/terms of the upstream 0xSero/Qwen3.6-28B and Qwen base models. (No explicit license file was present in the source; defer to upstream.)

Recommended config (tested)

Setting	Value	Why
`enable_thinking`	False for coding/tool-use/chat	Far more reliable & faster; thinking-ON can ramble.
Thinking budget	~6000 (when thinking ON)	Caps the reasoning loop so it commits instead of spiraling.
Sampling	temp 0.3–0.4, topp 0.9, topk 20	(temp ~0.7 for open chat)
`fix_mistral_regex`	True	Required for this quant's tokenizer.

Two good modes: (1) coding/tool-use → enable_thinking=False (fast, reliable); (2) hard open-ended reasoning → thinking ON + max_tokens ~5000–6000.

⚠️ Known incompatibilities (oMLX speculative features)

Because of the hybrid linear-attention/MoE architecture, these crash and should be left OFF:

—DFlash → [convert] Only length-1 arrays can be converted to Python scalars (in target_qwen_gdn). Also single-stream.
—SpecPrefill → '_OffsetAdjustedRoPE' object has no attribute 'dims'.
—MTP → this quant does not carry the mtp.* tensors.

Plain (non-speculative) decoding works great: ~78 tok/s in chat, ~49 tok/s on large (27K-token) agentic prompts. TurboQuant KV cache @ 6-bit is fine if you want to save runtime memory on long contexts.

Capability (informal, correctly-prompted)

Strong at: math, everyday programming, algorithms (LIS, Dijkstra, DP, thread-safe queues), tool-calling (clean JSON), and agentic planning. Weakness: niche factual recall (e.g. specific geography) — it can hallucinate and ruminate when it lacks knowledge; pair it with a search tool for fact-lookups.