Modelpublic

jclyons52/Qwen3.8-27B-UD-Q2_K_XL-MLX-kq-direct

sourceHugging Faceapache-2.0updated 29d agoView on Hugging Face

0likes2.2kdownloads

Model Card

Qwen3.8-27B UD-Q2KXL — MLX native IQ-quant (experimental)

2-bit dynamic IQ-quantized MLX build of Qwen3.8-27B, replicating unsloth's UD-Q2KXL GGUF bit-map (2.72 bpw, 9.5 GB) — quantized directly from the fp16 base with native non-linear IQ codecs (IQ1S/IQ1M/IQ2XXS/IQ2XS/ IQ2S/IQ3XXS/IQ3S/IQ4XS + K-quants) via mlx-kquant.

This is an experimental format. It requires mlx-kquant (which patches mlx_lm.load) and does not load with stock mlx-lm.

Per-tensor codec map (498 tensors)

codec	tensors	notes
iq3_xxs	112	attn_out, some ffn
q8_0	96	linear-attn a/b projections (kept high)
iq3_s	57	attnqkv, attngate, out_proj
iq2_s	67	ffn gate/up/down
iq2_xxs	48	ffn gate/up/down
iq2_xs	34	ffn gate/up/down
q4_k	21	v/k projections, outproj, attnoutput
iq1_s	20	first-block ffn gate/up, deep ffn
iq4_xs	19	out_proj, late ffn
q2_k	16	token embedding, some ffn
q3_k	5	misc
q5_k	2	v_proj (last layer)
iq1_m	1	layer-14 inprojqkv

Importance-weighted encoding uses unsloth's published imatrix (GGUF format, in_sum2/counts), with out-projection vectors un-permuted from llama.cpp's tiled V-head order back to HF grouped order.

Usage

bash

pip install mlx-kquant

python

from mlx_kquant.mlx_lm_patch import patch_mlx_lm_load
patch_mlx_lm_load()  # must run before mlx_lm.load

from mlx_lm import load
model, tokenizer = load("jclyons52/Qwen3.8-27B-UD-Q2_K_XL-MLX-kq-direct")

response = generate(model, tokenizer, prompt="Hello", max_tokens=100)

Quality (wikitext-2 first-32k, MLX harness, 512-token windows)

build	bpw	PPL
fp16 (MLX)	16	8.46*
affine Q2 map (our earlier build)	4.36	9.42*
this build	2.72	10.73*

*window-0 numbers on the MLX harness for apples-to-apples; full-32k PPL for this build is ~14.8. The GGUF original measures 7.18 under llama.cpp's perplexity tool, but llama.cpp and MLX perplexity numbers are not directly comparable (~1.3 ppl harness offset measured on fp16).

Honest summary: usable, coherent output at 2.72 bpw, but not yet at llama.cpp's IQ2 quality — the ggml encoder inside mlx-kquant is older than the one unsloth used. See RESUME-NOTES.md in the ud2mlx repo for the parity chase.

Build provenance

—Base: unsloth/Qwen3.8-27B fp16 (local snapshot, 2026-08)
—Codec map + imatrix: unsloth UD-Q2KXL GGUF artifacts
—Encoder: mlx-kquant (ggml CPU IQ encoders, Metal dequant/matmul)
—Linear-attn tensors (Alog, dtbias, conv1d, V-head projections) all from the fp16 base in HF head order — never from the GGUF (different base revision; see repo notes).