Modelpublic

spicyneuron/GLM-5.1-MLX-3.6bit

sourceHugging Facemitupdated 4mo agoView on Hugging Face

1likes57downloads

Model Card

GLM 5.1 optimized to run comfortably on a Mac Studio M3 512. This is the balanced version. Alternatives: speed-first, quality-first

—A mixed-precision quant that balances speed, memory, and accuracy.
—3-bit baseline with important layers at 4, 8 and BF16.
—Fits into ~350 GB memory, leaving plenty of room to run parallel models (ex: Minimax M2.7, Qwen 3.6 35B).

Usage

# Start server at http://localhost:8080/chat/completions
uvx --from mlx-lm mlx_lm.server \
  --host 127.0.0.1 \
  --port 8080 \
  --model spicyneuron/GLM-5.1-MLX-3.6bit

Benchmarks

metric	baa-ai/GLM-5.1-RAM-270GB-MLX	2.9 bit	3.6 bit (this model)	4.5 bit
bpw	3.110	2.906	3.645	4.538
base memory	269.303	251.702	315.648	392.992
peak memory (1024/512)	291.257	272.358	341.020	424.067
prompt tok/s (1024)	194.958 ± 0.075	194.216 ± 0.167	190.508 ± 0.880	193.563 ± 0.094
gen tok/s (512)	21.381 ± 0.050	19.527 ± 0.035	17.873 ± 0.156	17.259 ± 0.032
kl mean\*	0.686 ± 0.054	0.268 ± 0.009	0.117 ± 0.004	0.048 ± 0.002
kl p95\*	1.478 ± 0.054	0.537 ± 0.009	0.236 ± 0.004	0.097 ± 0.002
perplexity	4.780 ± 0.020	4.118 ± 0.016	3.945 ± 0.016	3.920 ± 0.016
piqa	0.776 ± 0.010	0.794 ± 0.009	0.820 ± 0.017	0.814 ± 0.017

\* GLM 5.1 KL divergence calculated against the largest quant I could run locally (~495 GB), so real KL is higher.

Tested on a Mac Studio M3 Ultra with:

mlx_lm.kld --baseline-model path/to/mlx-full-precision
mlx_lm.perplexity --sequence-length 2048 --seed 123
mlx_lm.benchmark --prompt-tokens 1024 --generation-tokens 512 --num-trials 5
mlx_lm.evaluate --tasks piqa --seed 123 --num-shots 0 --limit 500

mlx_lm.kld is approximate, based on top_k not full logits. Here's the code.

Methodology

Quantized with a mlx-lm fork, drawing inspiration from Unsloth/AesSedai/ubergarm style mixed-precision GGUFs. MLX quantization options differ from llama.cpp, but the principles are the same:

—Sensitive layers like MoE routing, attention, and output embeddings get higher precision
—More tolerant layers like MoE experts get lower precision