Brooooooklyn/Qwen3.6-27B-UD-Q8_K_XL-mlx

Qwen3.6-27B — UD-Q8KXL (mlx-node)

8-bit base mixed-precision quantization of Qwen/Qwen3.6-27B for Apple Silicon, using the **Unsloth Dynamic** quantization strategy via mlx-node.

All Variants

Benchmarked on Apple M3 Max 128GB via `examples/lm.ts` (best decode tok/s across turns 2–4, steady-state).

Performance

Steady-state decode: 9.9 tok/s on Apple M3 Max 128GB (best of turns 2–4, examples/lm.ts capitals chat with reasoningEffort: 'low'). Decode is memory-bandwidth bound on Apple Silicon — fewer bytes per token directly translates to higher throughput.

Per-Tensor Bit Assignments (N=8)

Quantization Strategy

Based on Unsloth Dynamic 2.0 per-tensor KLD analysis. Sensitive layers get higher bits with AWQ correction, while the bulk of FFN weights are aggressively quantized. imatrix AWQ pre-scaling amplifies important weight channels and fuses inverse scales into preceding layer norms (zero inference overhead).

AWQ-correctable projections (q/k/v, inprojqkv/z) are quantized at 8-bit via input_layernorm. Non-AWQ-correctable projections (oproj, outproj) are kept at bf16 — their inputs come from attention/GDN computation, not from a norm layer.

Architecture

Usage

import { loadSession } from '@mlx-node/lm';

const session = await loadSession('./Qwen3.6-27B-UD-Q8_K_XL-mlx');

for await (const event of session.sendStream('Explain the hybrid attention mechanism in Qwen3.6.', {
  config: { maxNewTokens: 2048, temperature: 0.6, reasoningEffort: 'low' },
})) {
  if (!event.done) process.stdout.write(event.text);
}

How It Was Made

mlx convert \
  -i Qwen3.6-27B \
  -o Qwen3.6-27B-UD-Q8_K_XL-mlx \
  -q --q-bits 8 --q-recipe unsloth \
  --imatrix-path imatrix_unsloth.gguf

Acknowledgments

• —[Unsloth](https://unsloth.ai) — Quantization strategy based on their per-layer KLD benchmarks and Dynamic 2.0 methodology
• —[Qwen Team](https://huggingface.co/Qwen) — For the Qwen3.6 model family
• —[Apple MLX](https://github.com/ml-explore/mlx) — For the Metal-accelerated ML framework

License

Apache 2.0 (inherited from base model).