CoolFace
Modelpublic

tacos4me/GLM-5.3-Flash-NVFP4-FP8ATTN-512K

sourceHugging Facemitupdated 27d agoView on Hugging Face
0likes451downloads
Model Card
[!WARNING] Preview (2026-08-31). The recipe and every number below are final and measured; the card is pending one last sign-off. Expect polish, not surprises.

GLM-5.3-Flash · 512k + MTP + vision, entirely in VRAM, on 2× RTX PRO 6000

524,288 tokens of context, MTP speculative decoding, and vision — simultaneously, with zero CPU offload — on two RTX PRO 6000 Blackwells (192 GB total, TP2, PCIe, no NVLink).

Three consecutive ~501,000-token prompts served to completion in 90–96 s each. 144.5 tok/s single-stream decode (MTP k=1 at 95.8% acceptance). Vision spot-set 10/10, before and after the 500k rounds. Teacher-forced quality 95.57% against the BF16-attention baseline — the same score this checkpoint gets running text-only (gate ≥95).

It fits because the ~19 GiB of BF16 the NVFP4 parent leaves behind — attention, shared/dense MLP, lm_head, vision tower — is re-quantized to block-FP8, and because the sparse-MLA KV cache drops from 656 to 444 bytes per token·layer: one fp8 + three fp6 tiles, minus a 128-byte RoPE slot this NoPE model never reads. The BF16-attention parent cannot load on this hardware without offloading experts to the CPU; this build serves half a million tokens on it with room left for a draft model and a vision tower.

Every number on this card was measured on this box. The raw dumps ship in `scripts/`.

TL;DR

**512k "everything" lane** 🌮**262k c=4 lane**
max context524,288262,144
concurrent streams1 (requests queue)4
decode, short ctx144.5 tok/s135.5 single · 289.2 aggregate @ c=4
speculative decode (MTP)k=1 (95.8% accept)k=1
multimodalvision on — 10/10 spot-setvision on — 10/10
largest served prompt501,093 tok (3/3 clean, 90–96 s)KV pool 473,550 tok (1.81× at 262k)
quality (TF top-1 vs BF16 base)95.57%¹ (gate ≥95)95.57%¹

Both lanes run with zero CPU offload. The offloaded setup they replaced decoded at 87 tok/s.

¹ At the 8666 compact-KV rung (1 fp8 + 3 fp6 tiles, 444 B/row). The KV compaction that fits MTP + vision at 524,288 costs zero measurable teacher-forced quality — the block-FP8 vision tower freed the weights (87.28 vs 87.47 GiB/rank) that pay for the fp8 first tile. The all-fp6 6666 rung measures 94.10% with a ~0.2 GiB thinner pin, if margin ever matters more than the last 1.5 points. Repetition probes at 251k: clean (worst 4-gram repeat 3, fail threshold ≥12).

[!IMPORTANT] Matched pair: this checkpoint does not load on stock vLLM or the vendor images. glm5_next lives in per-model images (vllm#53906 still open) which construct attention unquantized and cannot load an FP8 lm_head. Use the patch set in `serving/` (Dockerfile included). Without it: a KeyError on weight_scale_inv or a vocab assert.

Quickstart

bash
docker build -t local/vllm-glm53:fp8attn-512k serving/
docker build -t local/vllm-glm53:everything serving/everything-lane/

The 512k "everything" lane (MTP + vision + 524,288 — the headline config):

bash
docker run --init --rm --gpus all --runtime nvidia --ipc=host --network host \
  --shm-size=32g --ulimit memlock=-1 --ulimit stack=67108864 \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
  -e VLLM_KVQ_TILES=8666 \
  -v /path/to/GLM-5.3-Flash-NVFP4-FP8ATTN-512K:/model:ro \
  local/vllm-glm53:everything /model \
  --served-model-name glm-5.3-flash \
  --host 0.0.0.0 --port 8000 \
  --tensor-parallel-size 2 \
  --max-model-len 524288 \
  --gpu-memory-utilization 0.95 \
  --kv-cache-memory 3390000000 \
  --kv-cache-dtype fp8 \
  --max-num-seqs 1 \
  --max-num-batched-tokens 1024 \
  --limit-mm-per-prompt '{"image":1,"video":0}' \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1}' \
  --kernel-config '{"enable_jit_warmup":false,"enable_cutedsl_warmup":false}' \
  --compilation-config '{"cudagraph_capture_sizes":[1,2]}' \
  --enable-prefix-caching \
  --no-enable-flashinfer-autotune \
  --enable-auto-tool-choice --tool-call-parser glm47 --reasoning-parser glm45 \
  --trust-remote-code

Capture size 2 matters: MTP decode steps are 2 tokens, and an uncaptured size runs eager — ~27 tok/s instead of ~144. Images go in standard OpenAI image_url content parts (data URLs supported).

VLLM_KVQ_TILES selects the compact-KV rung:

rungrowmeasuredwhen
8666444 B6,364 B/tok · TF 95.57%the default above
6666416 BTF 94.10%~5% thinner KV pin, if margin is tighter than quality
8888528 Bbit-identical to stock rowsprovable-zero KV delta, ~15% more KV/token
unset656 Bpool 417,631 tokstock rows; caps this feature set at 327,680

262k c=4 concurrency lane — the same command with --max-model-len 262144 --max-num-seqs 4 --max-num-batched-tokens 4096 --compilation-config '{"cudagraph_capture_sizes":[1,2,3,4,6,8]}'. Captures must reach 8: MTP decode is 2 tokens per running sequence, so c=4 decode batches hit 8. Measured: c=1 135.5 tok/s, c=4 289.2 tok/s aggregate (72.3/stream, 2.1×); KV pool 473,550 tokens; vision 10/10.

Max-quality text-only variant at 512k (TF 95.57%, 101.7 tok/s) — image local/vllm-glm53:fp8attn-512k; drop -e VLLM_KVQ_TILES, --speculative-config, and vision: --kv-cache-memory 4241026048 --limit-mm-per-prompt '{"image":0,"video":0}' --compilation-config '{"cudagraph_capture_sizes":[1]}'.

Margin disclosure. Min-free watermark during a full 512k prefill sampled 110–290 MiB across boots and rungs, with recoverable allocator retries at the deepest dips — by design, and with zero request failures across every soak and round. Do not co-locate other GPU tenants during long prefills.

<details> <summary><b>Why every flag is load-bearing</b> (what makes 512k + MTP + vision fit)</summary>

  • —`--kv-cache-memory` (pinned) — the flag that makes the rest possible, and a disclosed trade: the pin bypasses vLLM's memory-profiling gate. On this GPU the boot-time free-memory reading is always ~93.89 of 94.97 GiB (the worker's own CUDA context ≈ 1.05 GiB counts as "used"), so --gpu-memory-utilization ≥ 0.989 can never boot. The explicit pin skips profiling and the util gate; 0.95 is a formality. Headroom was then proven by real ~501k soaks, not a profiler.
  • —`VLLM_KVQ_TILES=8666` (compact KV rows) — the sparse-MLA fp8_ds_mla KV row is 656 B/token: a 512 B fp8 latent plus a 128 B RoPE slot that is provably inert on this NoPE model (qk_rope_head_dim=0). The patch drops the dead slot and tiles the latent as one fp8 + three fp6 blocks → 444 B/row, 6,364 B/tok effective (measured). That is what lets a 532k-token pool plus the MTP draft's KV live in a 3.16 GiB/rank pin — canonical rows would need ~4.4 GiB this card does not have. Rows are re-staged to canonical 656 B on read; the attention cubin itself is untouched.
  • —`--max-num-seqs 1 --max-num-batched-tokens 1024` — the sparse-MLA prefill carries an O(context) transient (~4+ GiB at 500k). seqs 2 / batched 2048 passes short-context quality and then OOMs at a real 503k prefill. Single-sequence is the 512k use case; concurrent requests queue. (Concurrency wants the c=4 lane, which halves context to pay for it.)
  • —`--compilation-config '{"cudagraph_capture_sizes":[1,2]}'` — MTP decode steps are 2 tokens; both sizes must be captured or decode runs eager (~27 tok/s). JIT warmups off and the bounded-memory indexer (everything-lane/indexer.py) buy the rest.

</details>

<details> <summary><b>What exactly is quantized</b> (and what stays BF16, and why)</summary>

Rebuilt from the pristine parent by scripts/convert_fp8attn.py + convert_lmhead.py + convert_fp8visual.py — CPU-only, deterministic, calibration-free, bit-for-bit reproducible (determinism experimentally verified). 549 tensors go BF16 → F8_E4M3 weight + FP32 weight_scale_inv block scales, dispatched via a MIXED_PRECISION manifest in config.json:

familyFP8 blocksizeround-trip err
routed experts (incl. MTP draft)NVFP4 — unchanged, LibertAIDAI's work163.3 GiB—
KDA q/k/v + b/fa/ga (34 layers)[32,32]6.46 → 3.23 GiB~2.3–2.4%
o_proj, all 46 layers[128,128]3.63 → 1.81 GiB~2.3%
MLA qa/kva/q_b (12 layers)[128,128]0.75 → 0.38 GiB0.12–0.16%
shared experts + dense MLP[128,128]2.86 → 1.43 GiB0.16%
lm_head[128,128]1.18 → 0.59 GiB2.64%
vision tower Linears (ViT mlp/qkv/proj + merger, 124 tensors)[128,128]1.02 → 0.51 GiB2.64–2.65%

Kept BF16 on purpose: kv_b_proj (MLA absorbs it to BF16 at runtime anyway), KDA f_b/g_b (the upstream authors are most explicit about delta-rule precision there), the sparse indexer, embed_tokens, the vision conv/norm remainder (patch_embed, downsample), norms, routers.

The MLA/MLP families measure near-lossless (0.12–0.16%) because base GLM-5.3 is natively block-FP8 — those BF16 tensors are dequantized copies of FP8 originals. The KDA projections sit at the ~2.3% E4M3 floor: a real numerics change on a precision-sensitive path, which is why the quality gate exists. We do not call the attention conversion lossless; we call it 95.57% teacher-forced, with protocol, n, and raw dumps published.

Block [32,32] for the fused KDA inproj: vLLM fuses q,k,v,b,fa,g_a into one module with a 32-row b shard at a non-128-aligned offset; the block must be square and divide 32.

Integrity: 121 shards, 150,651 tensors, index↔header cross-check zero mismatches; total_size exact (186,149,192,912 B = 173.36 GiB).

</details>

<details> <summary><b>Measured evidence</b> (memory ledger · fit ladder · quality gates · kernel story · one honest negative)</summary>

All numbers measured on 2× RTX PRO 6000 Blackwell Workstation (96 GB, TP2, PCIe), vLLM 0.1.dev20051+g487ecf187 in the patched image. Raw dumps ship in scripts/.

Memory (per rank, vLLM's own ledger). At identical flags to the old offloaded config, FP8 attention+MLP moved the KV pool 340,163 → 645,997 tokens (+90%) and max loadable context 315,392 → 524,288. Offload-free at 524,288: weights+non-torch 84.77 GiB/rank (MTP-off −2.01, vision-skip −0.53, FP8 lm_head −0.29 — each landed to the decimal), KV pinned 3.95 GiB = 526,825 tokens at fp8 (~7.84 KiB/token/rank).

The fit ladder (how the flags were found):

#attemptresult
Autil 0.985, seqs 2, b2048, MTP/mm offFAIL — avail 3.08 vs 3.92 GiB; O(maxlen) profile peak
Butil 0.99 + trimsFAIL — util ≥0.989 can never boot (own CUDA context in the reading)
B3`--kv-cache-memory` pinboots + serves; pool 552,543 tok
C1+ tuned kernels, seqs 2quality PASS, 101.7 tok/s — OOM at a real 503k prefill (O(context) indexer transient)
C2seqs 1, b1024, pin 3.95, graphs [1]ACCEPTANCE PASS — 3×503k clean, watermark 290 MiB, zero failures
D+ MTP + vision at 524,288, stock KV rowscannot fit — draft KV alone wants ~1.5 GiB the card doesn't have
D2+ `VLLM_KVQ_TILES=6666`, pin 3.07ACCEPTANCE PASS — every gate on one boot; TF 94.10%
E+ block-FP8 vision towerweights 87.28 vs 87.47 GiB/rank; vision 12/12 = BF16 baseline
E2+ rung `8666`, pin 3.16TF back to 95.57% — the freed weights fund the fp8 first tile

The everything lane at 524,288 (acceptance boot + rung-promotion confirmations): KV pool 547,486 tokens from a 3.16 GiB/rank pin (6,364 B/tok measured vs 6,352 predicted from the row math); MTP active at 95.8% acceptance; 144.5 tok/s median short-context decode; vision 12/12 objective probe + 10/10 spot-set, re-probed clean after the 500k rounds; repetition probes at 251k clean (worst 4-gram 3); fresh ~501k-token prefills in 85–96 s, zero hard OOMs across 5 rounds and 3 boots; min-free watermark sampled 110–270 MiB (boot-variance-dominated, recoverable allocator retries at the deepest dips). The rung ladder is exact: all-fp6 6666 fits with the fattest margin at 94.10% TF; 8666 recovers 95.57%, paid for by the FP8 vision tower's freed weights; bit-identical 8888 would need ~0.5 GiB more pin than exists.

Quality gates (protocol: teacher-forced, identical context, n=271 scored positions; plus 10 greedy generations; scored against the unmodified NVFP4 parent under the unpatched vendor image — reproduce with scripts/compare.py):

metricMLP-only FP8+ attention FP8pre-recal**shipped (recal scales)**everything lane (8666 KV)
teacher-forced top-197.4%97.05%96.68%95.57% (gate ≥95)95.57%² (gate ≥95)
greedy agreement to 1st divergence94.7%90.5%60.0% (n=25, noisy)¹90.82%—
long-gen degeneracynonenonenonenone, incl. 2,712-tok gens at 225k & 259,632 ctxnone, incl. 2,912-tok gens at 251k ctx

¹ Greedy chains on a reasoning model diverge at the first flipped token; teacher-forced is the signal. ² At the 8666 KV rung + block-FP8 vision tower (scripts/results-tf-8666.json). The all-fp6 6666 rung measures 94.10% (96.31% agreement vs the text-only lane's own dump) — disclosed, selectable by env when a thinner KV pin matters more than the last 1.5 points.

The patched image is bitwise neutral for BF16-attention checkpoints (100.00% agreement, |Δlogprob| = 0.00000 vs the vendor image). MTP acceptance stays 81–98%. 17×23 → 391 on every configuration, including served at the 524,288 ceiling.

The kernel story. vLLM ships zero SM120-tuned w8a8 block-FP8 triton configs. The fused KDA inproj (N=12576, K=4096, block [32,32]) ran a default config at 313 µs/call — **43.8% of decode GPU time** going to one GEMM. One autotuned JSON (ships in `serving/configs/`, 22.3 µs at M=1) plus dropping CPU offload took the checkpoint 66 → 142 tok/s at 114k. Same `BLOCKSIZE_K` k-split, so numerics unchanged. The block-128 GEMMs already ran vendored DeepGEMM SM120 kernels at ~1.35 TB/s and were left alone.

One honest negative. MLP-only FP8 (round 1) was a net loss: vLLM reserves the profiled activation peak, and activating the FP8 block-GEMM path costs a fixed +0.80 GiB/rank regardless of layer count — it outspent the 0.71 GiB saved (pool −2.8%). Only when attention (+2.81 GiB/rank) joined did the fixed cost amortize 4:1. If you are building a partial-FP8 derivative: go all-in or don't bother.

Input-scales provenance. The parent's expert input_scale tensors went through three revisions in one day: an input_scale=1.0 placeholder (357b45cc), its retraction (discussion #7 — small-amax blocks underflow to zero, input-dependent repetition), and a same-day recalibration (caca4e6a). This repo pins `caca4e6a` and gates it directly: 95.57% TF (gate ≥95), repetition probes clean at 225k and 259,632-token contexts (worst 4-gram repeat 3 and 2 vs fail ≥12), a 503,374-token round in 80.4 s. Dumps: scripts/results-recal.json, results-longrep-recal.json, results-longrep-prod-topup.json.

</details>

Credits

  • —[LibertAIDAI/GLM-5.3-Flash-NVFP4](https://huggingface.co/LibertAIDAI/GLM-5.3-Flash-NVFP4) — the NVFP4 expert quantization and input scales this build stands on.
  • —[chriswritescode-dev/glm-5.3-flash-sm120](https://github.com/chriswritescode-dev/glm-5.3-flash-sm120) — the SM120 image lineage serving/ patches against; and the vLLM project (Apache-2.0).
  • —[local-inference-lab](https://github.com/local-inference-lab/vllm) — independently landed the equivalent MLA quant-config passthrough in their fork (`8590bf9c`) around the same time.
  • —[brandonmusic/GLM-5.3-Flash-tr3-4bpw](https://huggingface.co/brandonmusic/GLM-5.3-Flash-tr3-4bpw) — first zero-offload long-context GLM-5.3 on this hardware (EXL3 route); worth a look if 4-bit KV trade-offs suit you.

Repo layout

├─ model-000{01..120}-of-00120.safetensors   # 173.4 GiB: NVFP4 experts + block-FP8 + BF16
├─ model-input-scales.safetensors            # recalibrated scales (parent @caca4e6a)
├─ config.json                               # MIXED_PRECISION manifest (38k+ entries)
├─ serving/                                  # matched-pair runtime patch (Apache-2.0, vLLM-derived)
│  ├─ Dockerfile ├─ kda.py ├─ model.py ├─ modelopt.py ├─ configs/*.json
│  └─ everything-lane/                       # + MTP/vision/512k patch set (compact KV, bounded indexer)
└─ scripts/                                  # conversion + acceptance harness + raw quality dumps

serving/ is the public cstechdev/vllm:glm53-flash-nope-sm120-cu130-20260826-r1 image (the chriswritescode-dev/glm-5.3-flash-sm120 overlay of the official per-model image) plus exactly four files: the KDA and MLA quant_config passthroughs, the FP8BLOCK dispatch + FP8 `lmhead loader, and the SM120-tuned triton config. serving/everything-lane/ layers the MTP + vision + 512k patch set on top: compact KV rows (VLLMKVQTILES), the bounded-memory sparse-MLA indexer, and the embedding/vision quant_config` passthroughs. Apache-2.0, SPDX headers retained; the base image is pulled from Docker Hub, not redistributed.

Known limitations

  • —Matched pair — loads only under the serving/ patched images.
  • —512k is single-stream (--max-num-seqs 1); the sparse-MLA prefill transient is O(context). Concurrency is the c=4 lane's job, at half the context.
  • —One image per prompt as configured (--limit-mm-per-prompt); the vision tower ships block-FP8 (12/12 objective probe, identical to its BF16 baseline). Video untested.
  • —110–290 MiB margin during a full 512k prefill (boot-variance-dominated) — no co-tenants.
  • —The `6666` rung trades quality for pin (94.10% vs 95.57%) — selectable, disclosed, off by default.
  • —SM120 only measured — the format is hardware-agnostic; the kernel selection and tuned config are not.

License

Weights MIT © 2026 Z.AI Co., Ltd (LICENSE verbatim). Chain: zai-org/GLM-5.3-Flash → LibertAIDAI/GLM-5.3-Flash-NVFP4 (caca4e6a; NVFP4 expert quantization and input scales are their work) → this repo (block-FP8 attention/MLP/lmhead/vision). `serving/*.py` Apache-2.0 (vLLM-derived). Conversion CPU-only, deterministic, calibration-free; `scripts/convertfp8attn.py + convertlmhead.py` + `convertfp8visual.py` reproduce the checkpoint bit-for-bit. Not affiliated with Z.ai / Zhipu or LibertAI.