Modelpublic

vcruz305/GLM-5.3-Flash-EXL3-K2

sourceHugging Facemitupdated 24d agoView on Hugging Face

4likes1.4kdownloads

Model Card

GLM-5.3-Flash EXL3 K2

Community EXL3 K2 pack of zai-org/GLM-5.3-Flash-BF16: 2-bit MCG trellis on routed MoE experts only. Attn / shared / embed / head / vision stay source-native. 120 safetensors shards, 97,728,721,536 B (91.017 GiB).

### These weights need a specific vLLM build (K-pool fix shipped 2026-08-30; reinstall) Stock vLLM cannot load this pack. It has neither the exl3 quantization method nor the Glm5Next architecture, and no flag turns them on. pip install vllm will appear to work and then fail after you have downloaded 91 GiB. Prebuilt wheels, no compiler, minutes to install: [vcruz305/GLM-5.3-Flash-EXL3-K2-spark-vllm](https://huggingface.co/vcruz305/GLM-5.3-Flash-EXL3-K2-spark-vllm)

Where	What
This repo	the weights, 120 shards, 91.017 GiB
spark-vllm	prebuilt vLLM + ExLlamaV3 wheels for GB10
recipe	install scripts, serve flags, full measurements


Format	EXL3, `bits=2`, codebook `mcg`, scope `glm53_routed_experts_only`
Architecture	`Glm5NextForConditionalGeneration`
Spark (TP=1)	native MTP k=2: 15.7–16.5 tok/s @ 8k; 14.6–15.7 tok/s @ 64k
Max ctx allocated	131072 (KV 786,432 @ util 0.91); recommended serving ctx 65536, see ladder
sixcat 0.5.1	120/120 think-on @ 64k — overall 84.17 on the fixed 2026-08-30 runtime (flagged: `mmlu:8` and `ifeval:1300` hit their budgets), same overall as the earlier container run
KLD vs BF16 (fidelity suite v1)	512 contexts, 1,048,064 positions, full-vocab KL through the shared head — token-mean 0.3346 nats (95% CI 0.320–0.349), median 0.117, p99 3.33, top-1 agreement 0.788; the suite's FP8 anchor is 0.028 / 0.943; the K2/K3 mix scores 0.3121 / 0.795 on the same contexts. Method and scorer validation in the recipe's `docs/KLD.md`
long context	boots at 262,144 (KV pool 1,093,332 tokens, 4.17x full-256k concurrency); real-text needle ladder verified to 163,479 prompt tokens (needle recalled verbatim, prefill ~590 tok/s, decode 17–20). Known runtime bug: prompts between ~163k and ~180k wedge the engine — keep prompts ≤163k until the fix lands; ≤131,072 contexts unaffected. Details in the recipe README
Also measured	4× L40 TP=4 text `/v1` smoke (see below)

This is a quantized weight pack. Pull it, then follow the GitHub recipe for vLLM flags. Weights are MIT (Z.AI).

Quick start

bash

# 1. the runtime first, so a bad environment fails in seconds instead of hours
git clone https://github.com/vcruz305/GLM-5.3-Flash-EXL3-K2-DGX-Spark-recipe
cd GLM-5.3-Flash-EXL3-K2-DGX-Spark-recipe
python scripts/preflight.py
bash scripts/install_prebuilt.sh

# 2. then the 91 GiB
hf download vcruz305/GLM-5.3-Flash-EXL3-K2 \
  --local-dir ~/models/GLM-5.3-Flash-EXL3-K2

--local-dir resumes. There is no --resume-download flag. Last path component must be the Hub basename.

One-Spark vLLM (winner flags). Full notes, ctx ladder, and scripts: [GLM-5.3-Flash-EXL3-K2-DGX-Spark-recipe](https://github.com/vcruz305/GLM-5.3-Flash-EXL3-K2-DGX-Spark-recipe).

bash

export EXL3_FUSED_MOE=1
vllm serve ~/models/GLM-5.3-Flash-EXL3-K2 \
  --served-model-name GLM-5.3-Flash-EXL3 \
  --quantization exl3 \
  --tensor-parallel-size 1 \
  --kv-cache-dtype fp8 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.87 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 2048 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
  --tool-call-parser glm47 --enable-auto-tool-choice --reasoning-parser glm45 \
  --skip-mm-profiling --limit-mm-per-prompt '{"image":4,"video":1}'

Load log must show fused exl3_moe at bits=2. Bits=2 loading is handled natively by the EXL3 plugin in the runtime wheels, so there is nothing to patch. Do not pass --moe-backend marlin. Native MTP; do not mix a DFlash sidecar on the same server.

Eval/serve at 64k: MAX_MODEL_LEN=65536 GPU_MEM_UTIL=0.91 (same MTP k=2). Spec A/B was ranked at 8k so MLA page size stayed in one band.

Spark measurements (2026-08-29)

Hardware: one NVIDIA GB10 (SM121), ~121 GiB unified. Engine: vLLM --quantization exl3, EXL3_FUSED_MOE=1. Bench: streamed /v1, thinking off, 128 gen, seqs=1.

Spec ranking @ 8k

Config	Decode tok/s	Accept
no spec	9.6–9.8	—
DFlash sidecar k=7	11.5	mean ~1.8 / 7
DFlash sidecar k=3 (older runtime)	12.8	mean ~1.8 / 3
DFlash k=3 BF16, Triton draft attn	15.6	26–79% by workload
MTP k=1	14.8	76–80%
MTP k=2	15.7–16.5	~74/44%, mean ~2.2

Context ladder (KV from engine log; decode = same 128-token bench when run)

max_model_len	spec	util	GPU KV	Decode
8192	none	0.87	192k–208k	9.6–9.8
8192	DFlash k=7	0.87	15,281	11.5
16384	DFlash	0.91	45,095	`/v1` pong
32768	DFlash	0.91	90,035	allocated (attn block 7168)
8192	MTP k=2	0.87	104,857	15.7–16.5
65536	MTP k=2	0.91	786,432	14.6–15.7
131072	MTP k=2	0.91	786,432	allocates; 81,920-token prompt OK, 98,304 faults

131072 allocates (KV 786,432, 6.00x) and an 81,920-token prompt completes, but a 98,304-token prompt faults in the EXL3 fused-MoE path and kills the engine, so 65536 stays the recommended serving ctx. DFlash at 8k / util 0.87 cannot climb ctx until util 0.91 (draft KV collapse).

sixcat 0.5.1 think-on, vendor glm-5.x, 20/category, host-guarded HumanEval, 64k serve: knowledge 65 · math 100 · truth 85 · instruct 75 · code 90 · tools 90. overall 84.2 with trunc-in-think:instruct — not a clean overall.

What is in the shards

—Routed experts: EXL3 K2 (suh, svh, trellis, (1,) int32 mcg) on gate_proj / up_proj / down_proj
—37,152 expert weights + source-native non-expert tensors
—model.safetensors.index.json, tokenizer, processor, generation config, chat template
—quantization_config.json: quant_method=exl3, bits=2, codebook=mcg, scope=glm53_routed_experts_only

Also measured: 4× L40 TP4

Text /v1 smoke on 2026-08-28, 4× NVIDIA L40 (SM89), TP=4, 8k, fused exl3_moe, no BF16 expert reconstruct. /health 200; 17 * 6 → 102. Build context under runtime/l40-vllm/ in this repo (CUDA 8.9, dense MLA fallback on L40). That gate is L40 TP4 text @ 8k. One-Spark GB10 flags and ctx live in the GitHub recipe.

GLM-5.3-Flash EXL3 K2

Quick start

Spark measurements (2026-08-29)

What is in the shards

Also measured: 4× L40 TP4

Links