Modelpublic

wfiedler/Qwen3.8-27B-ABLITERATED-mlx-q3km

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face

1likes401downloads

Model Card

Qwen3.8-27B-ABLITERATED-mlx-q3km

MLX conversion of Blackfrost-AI/Qwen3.8-27B-ABLITERATED-BF16, quantized with the mixed_3_4 recipe as an equivalent to GGUF Q3_K_M.

—3.910 bits per weight, 13.4 GB across 3 safetensors shards
—Architecture qwen3_5 (Qwen3_5ForConditionalGeneration), multimodal, hybrid linear/full attention every 4th layer
—Requires mlx-vlm >= 0.6.13 (earlier versions have no qwen3_5 module)

Why `mixed_3_4` and not `--q-bits 3`

MLX's mixed-bit predicates are a deliberate port of llama.cpp's _K_M scheme, not an approximation. Both give more bits to v_proj and down_proj in the first eighth of layers, the last eighth, and every third layer between, plus a high-bit lm_head. mixed_3_4 is 3-bit base with 4-bit on those sensitive tensors, which is exactly Q3KM.

The measured result confirms it: 3.910 bpw here, against 3.91 bpw for the official GGUF Q3KM (13.30 GB for ~27.2B params). A plain --q-bits 3 would land near 3.5 bpw and would not match.

Note that mlx-vlm never quantizes the vision encoder at any level, so the vision tower stays bf16.

Conversion

bash

mlx_vlm.convert \
  --hf-path Blackfrost-AI/Qwen3.8-27B-ABLITERATED-BF16 \
  --mlx-path ./Qwen3.8-27B-ABLITERATED-mlx-q3km \
  --quantize --quant-predicate mixed_3_4

Usage

bash

pip install -U mlx-vlm

# text
mlx_vlm.generate --model wfiedler/Qwen3.8-27B-ABLITERATED-mlx-q3km \
  --prompt "Explain unified memory in two sentences." --max-tokens 256

# vision
mlx_vlm.generate --model wfiedler/Qwen3.8-27B-ABLITERATED-mlx-q3km \
  --prompt "Read this receipt and list every line item with its price." \
  --image receipt.png --max-tokens 400

Quality vs the 5-bit conversion

Benchmarked against the q5 build (5.678 bpw, 18 GB) of the same base model, on images with exact ground truth, 12 tasks x 3 trials at temperature 0.7.

task	tests	q5	this model
receipt	OCR, 14 fields	14/14 x3	14/14 x3
chart	bar values	6/6 x3	6/6 x3
table	5x5 numeric cells	8/8 x3	8/8 x3
small text	small-font rare tokens	6/6 x3	6/6 x3
shapes (10)	count 3 groups	6/6 x3	6/6 x3
shapes (20)	count 4 scattered groups	3/3 exact	0/3 exact
photos	species ID	3/3	3/3
signs	real-world OCR	2/3 + 1 loop	3/3
logic / recall	text reasoning	3/3, 3/3	3/3, 3/3

On reading images the two are indistinguishable. Every OCR-shaped task ties perfectly: all receipt fields, all chart values, all table cells, every rare token in the small print.

The one real gap is counting many objects. Given 20 scattered shapes in four colour/type groups, q5 answered correctly three times out of three; this model answered 17, 19 and 21, never correct. The difference is visible in the output: q5 enumerates shape by shape before totalling, while the 3-bit build emits a terse number. The low-bit quant lost the habit of decomposing before answering.

Neither model does reliable mental arithmetic without a scratchpad; both compute a four-item subtotal wrongly at temperature 0.7.

	q5	this model
on disk	18 GB	12 GB
bits per weight	5.678	3.910
weights resident	20.68 GB	14.11 GB
peak during image generation	23.24 GB	16.67 GB
generation	12.9 tok/s	18.9 tok/s

Measured on an Apple Silicon Mac with 64 GB unified memory, each model in its own process. (GenerationResult.peak_memory is a process-global high-water mark that does not reset between models, so comparing two in one process makes the second inherit the first's peak.)

Use this build for OCR, document and chart work — 46% faster and 6.6 GB lighter for no measurable loss. Use the 5-bit build when the task involves counting or enumerating things in a scene.

Provenance and caveats

This is an abliterated model: the base has had refusal behaviour removed by its authors. It will comply with requests a stock instruct model would decline. Evaluate it before using it anywhere user-facing.