hermia/glm-5.2-sm120-recipe

GLM-5.2-504B (NVFP4) on 4× RTX PRO 6000 — 250K context, stock vLLM

This is a recipe, not a model. No weights here. It documents how to serve `0xSero/GLM-5.2-504B-Nvidia` at 250,000-token context on 4× RTX PRO 6000 Blackwell (sm_120) using stock vLLM 0.25.1 — no fork, no Docker, no CUDA 13.2, no b12x.

👉 Full recipe, patch, and scripts: https://github.com/hermia-ai/glm-5.2-sm120-stock-vllm

Measured results

All measured on real hardware, 2026-07-15. Nothing here is estimated.

The three walls between you and a working server

Each is documented in the GitHub repo with the exact error string and file:line cause:

1. 1.OOM at 93.64 GiB/GPU. An upstream transformers bug: GlmMoeDsaConfig.__post_init__ unconditionally re-routes a legacy num_experts key over an explicit n_routed_experts. REAP-pruned checkpoints ship both keys, so vLLM builds FusedMoE for 256 experts that do not exist (log: Local/global number of experts: 64/256, correct is 42/168). Fix: delete the stale `num_experts` key from `config.json`.
2. 2.`TypeError: trtllm_batch_decode_with_kv_cache_mla() got an unexpected keyword argument 'kv_scale_format'`. vLLM 0.25.1 pins flashinfer-python==0.6.13 but its own sm120 sparse-MLA call site passes a 0.6.14-only kwarg. Fix: install 0.6.14 + FLASHINFER_DISABLE_VERSION_CHECK=1.
3. 3.`AssertionError: Decode Context Parallelism (DCP) requires attention implementations to return the softmax LSE during decode, but FlashInferMLASparseSM120Impl does not.` — and it tells you to pick another backend when that is the only backend on sm_120. Fix: a ~126-line patch (an independent reimplementation of vLLM PR #47779).

What does NOT work (documented so you don't repeat it)

• —MTP / speculative decoding — the drafter eats ~6.96 of the 6.97 GiB KV budget → estimated maximum model length is 256. MTP and 250K context are mutually exclusive on 4×96 GB.
• —`0xSero/GLM-5.2-504B-W4A16` (tempting: 57 GB smaller) — cannot load. indexer.wk ships int4-packed but vLLM hardcodes that layer quant_config=None → KeyError. Unreachable by any flag.
• —SGLang — needs bf16 KV on sm_120, which doubles per-token bytes vs fp8. A regression.
• —Raising `--max-model-len` without DCP — the KV pool is a fixed byte budget; it does not grow.

Credit

The DCP patch is an independent reimplementation of [vLLM PR #47779](https://github.com/vllm-project/vllm/pull/47779), not an improvement on it — the design credit is upstream's. This work contributes validation on real sm_120 silicon.

Thanks to [0xSero](https://huggingface.co/0xSero) for the checkpoint and for glm-5.2-sm120 — the complementary fork/Docker path, which reaches ~710K KV and MTP=3 at the cost of a multi-repo source build. If you want those numbers, use theirs; if you want to stay on stock vLLM, use this.

Also: DeepGEMM PR #324 (NVIDIA DevTech — sm120_fp4_mqa_logits), FlashInfer PR #3395 (SM120 sparse MLA), [lukealonso/b12x](https://github.com/lukealonso/b12x).

Licensed Apache-2.0.