0xdfi/GLM-5.2-1M-context-NVFP4-4x-DGX-Spark
GLM-5.2 at 1M context on 4× DGX Spark — our NVFP4 long-context recipe
*We wanted longer context than anything else could deliver for GLM-5.2 on this hardware — so we built our own recipe. Unpruned GLM-5.2 (744B / ~40B active) on a 4× NVIDIA DGX Spark (GB10, `sm_121a`) cluster, reaching up to 1,000,000 tokens of context by pairing an NVFP4 368-byte compact KV cache (58% the size of FP8) with every current leading speed-efficiency lever*: B12X sparse-MLA attention, MTP-5 speculative decode, full CUDA graphs, Marlin atomic-add, and concurrency-aligned graph capture.
Every number below is read live from vLLM's own engine logger — measured, not modeled, not benchmark-gamed.
⚡ Speed per version (live-measured)
Single-stream peaks at ~42 tok/s — the 4×-Spark hardware ceiling (independently confirmed by three rigs). v1.1 concurrency serves 4 simultaneous users at 103.7 tok/s aggregate (~2.5×). Decode is MTP-acceptance-bound: 42 peak on structured content, ~30 at 64K, ~22 floor on adversarial content.
Operating-point matrix (measured 2026-07-20, full CUDA graphs, seqs=1). Two knobs trade off: prefill batch (bigger = faster prefill, less context — indexer scratch = ctx×batch) and DCP level (more sharding = slower prefill+decode, more context). Decode is batch-independent (acceptance × forward-rate).
Max context scales sub-linearly with DCP (index-cache not sharded). No swap / no NVMe thrash at any point. Full sliding-scale matrices (context × prefill by batch, per DCP) at the top of the GitHub README.
vs the field: the best public 4×-Spark GLM-5.2 report is ~20–22 tok/s single-stream — and it pruned the model to get there. This runs unpruned at 42 peak / 104 aggregate.
⚡ Prefix caching — repeated prefixes skip prefill (on by default, v1.2)
Automatic prefix caching is enabled by default in the launcher. Requests sharing a leading prefix — multi-turn chat re-sending history, a shared system prompt across users, the same document re-queried — reuse cached KV and skip prefill for the shared span. Measured live (DCP2, identical ~25.6K-token prompt sent twice, single stream): 45.9s → 0.78s (≈59× faster), byte-identical output. Each hit saves ≈2s per 1,000 cached prefix tokens. Verified safe on this stack (B12X_MLA_SPARSE + nvfp4_ds_mla + DCP2 + MTP-5) — it does not raise peak KV memory (cached blocks are LRU-evictable). Set ENABLE_PREFIX_CACHING=0 only for clean prefill benchmarking. Helps when the varying text is at the end of the prompt; nothing for all-unique prompts.
Why it works
Reproduce
Full serve command, parameterized launcher, benchmark harness, findings, and the concurrency patch are in the companion repo — including the exact image, env, and DCP1/2/4 configs:
➡️ https://github.com/0xdfi/GLM-5.2-1M-4x-DGX-Spark
Model checkpoint served: `QuantTrio/GLM-5.2-Int4-Int8Mix` (unpruned, stock — not redistributed here). Runtime: custom vLLM fork exp1sm121a368r4dtypefix.
Honest limitations
- Decode is modest by datacenter standards (~22–42 single-stream) — that's Spark's ~273 GB/s bus, not the stack. For raw tok/s, Spark is the wrong tool; for 1M context on cheap silicon, it's remarkable.
- 63% of every decode step is collective/launch overhead (measured) — the real remaining ceiling lever is an RDMA one-shot allreduce, not a better drafter.
- Custom fork required — stock/nightly vLLM doesn't serve 744B @ 1M on
sm_121atoday. - Numbers are from one rig. Reproduce on yours before quoting.
Acknowledgments
Built on open community work — above all CosmicRaisins (the sm_121 sparse-MLA port + Triton kernels the whole stack depends on), tonyd2wild (the 200K recipe + Speed-Night optimization audit), QuantTrio (the checkpoint), zai-org (GLM-5.2), and ciprianveg / Zatz / back199640 / p33zy / aidendle94 / eugr / drowzeys / jasl plus the NVIDIA developer forum 374125 community. Full attribution in NOTICE. Apache-2.0.
Measured 2026-07-20 on 4× DGX Spark / GB10 sm_121a. Read from the live engine, not modeled.