jarrelscy/GLM-5.2-NVFP4-AQLM-hybrid-500k

GLM-5.2-Vision NVFP4+AQLM hybrid (500k context variant)

The 524,288-token context variant of GLM-5.2-Vision-NVFP4-AQLM-hybrid, now with image input, for 4x 96 GB SM120 GPUs. The smaller KV cache and attention workspace fund 48% of experts at NVFP4 (4.5 bpw, covering ~68% of routed computations); the rest are 2-bpw AQLM. Weights: 334 GB text + ~912 MB BF16 vision head. Trades context for more NVFP4 → higher text quality than the 1M sibling at a smaller window.

Modality — vision + text. This repo grafts the same Baseten GLM-5.2-Vision head as the 1M flagship (MoonViT tower + PatchMerger projector, byte-identical BF16) onto this variant's 48% NVFP4 text backbone. Serving now requires the `vision-graft` vLLM branch and --trust-remote-code (see below) — the text-only cudagraphs-v2 branch cannot load this repo's Glm5vForConditionalGeneration config. Text-only prompts behave exactly as before; the graft leaves the text path untouched. Checkpoint currency (2026-07). Expert weights are at parity with the 1M flagship's latest quantization generation: AQLM codebooks and scales were refreshed on every hybrid layer, and the deepest layers were re-tiered to the flagship's hot/cold NVFP4 allocation.

Credits

• —[Baseten](https://huggingface.co/baseten) built the vision graft (baseten/GLM-5.2-Vision-NVFP4): the MoonViT vision encoder from Moonshot AI's Kimi-K2.6 bolted onto Z.ai's GLM-5.2 through a PatchMerger projector Baseten trained (49.5M params, 1152→4608→6144) — without touching a single GLM weight. The vision tower (MoonViT-3d, 27 layers, 1152-dim, ~417M params) and projector here are byte-identical BF16 copies of Baseten's release (the same files shipped in the 1M repo).
• —Z.ai — GLM-5.2 (MIT). Moonshot AI — Kimi-K2.6 / MoonViT (Modified MIT).

🐳 Serve on SM120 (recommended)

Use the vLLM fork jarrelscy/vllm-glm52-sm120 — the hybrid NVFP4+AQLM MoE support, SM120 sparse-MLA (DSA) fixes, DCP, native MTP speculative decode, and the `glm5v` vision graft are all already committed (no runtime patching). Note the branch: image input lives on `vision-graft`, not the text-only cudagraphs-v2 branch.

Decode throughput is set by the config, not the checkpoint — all variants decode identically per-config (validated on the 1M sibling: needle-in-haystack PASS @ 749K). The variants differ only in context ceiling; this checkpoint tops out near ~500k, so use MAXLEN=500000.

git clone -b vision-graft https://github.com/jarrelscy/vllm-glm52-sm120
cd vllm-glm52-sm120
docker build -f Dockerfile.glm52-sm120 -t glm52-vision-sm120 .

# ★ PREFERRED — ~500k window WITH lossless MTP spec, TP speed + graphs + vision.
#   The tp4-1m-mtp entrypoint applies the promoted stack automatically
#   (IndexShare + bit-exact gemv kernels + ag_rs/NCCL-P2P + chunk 4096
#   / util 0.97); MAXLEN caps the window at ~500k here.
docker run --gpus all --ipc=host -p 8001:8001 \
  -v /path/to/weights:/models/1m:ro \
  -e PARALLEL=tp4-1m-mtp -e MAXLEN=500000 -e UTIL=0.97 \
  glm52-vision-sm120 --trust-remote-code

Three things differ from the text-only card and all three are required:

1. 1.Branch `vision-graft` (built on cudagraphs-v2). It adds the glm5v architecture (Glm5vForConditionalGeneration): MoonViT tower + projector wired into the fork's in-tree Kimi-K2.5 vision code, with the language model swapped to the hybrid GlmMoeDsaForCausalLM stack (DCP + native MTP + SM120 sparse-MLA fixes
2. 2.NVFP4/AQLM MoE kernels). The text-only branch cannot load this repo's config.
3. 3.`--trust-remote-code` — image preprocessing uses the Kimi-K2.5-style processor files shipped in this repo (kimi_k25_processor.py etc.).
4. 4.`MAXLEN=500000` — this variant is sized for a ~524K window; the tower+projector add ~58K tokens of KV pool, so the served window is capped here rather than at the 1M repo's 950K.

Reproducible build: this image pins flashinfer to a known-good commit and forces a NotImplementedError stub for nvidia-cutlass-dsl's cute.experimental (the 4.5.2 wheel was republished in place with a CuteExperimentalDSL ref that fatally AttributeErrors at import flashinfer.comm on CUDA<13.1). A plain docker build is therefore index-independent.

Then send images through the standard OpenAI API:

curl http://localhost:8001/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "glm-5.2",
  "messages": [{"role": "user", "content": [
    {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}},
    {"type": "text", "text": "Describe this image."}
  ]}]
}'

Performance (measured, 4x RTX PRO 6000 SM120, 2026-07-13 promoted stack)

Text-path numbers carry over unchanged (the graft leaves the text path untouched):

• —Single-stream decode (`tp4-1m-mtp`): short context ~77 tok/s counting / 71 code / 54 prose; at 123K context ~49 / 45 / 36 tok/s. Lossless (64K temp-0 golden-gated; reasoning canary PASS). This variant is slightly faster than the 1M sibling (73/64/52 short, 44/41/32 @123K) because its 48%-NVFP4 mix puts more experts on the fast path.
• —Fresh prefill: ~0.95K tok/s @123K (chunk 4096 + ag_rs/NCCL-P2P transport); prefix caching amortizes it across requests.

tp4-1m-mtp gives ~1M+spec on the 1M sibling; here the same config gives ~500k+spec (DCP — a stock vLLM feature for MLA models — shards the MLA latent KV by sequence; MTP's MLA-shaped draft KV shares that sharded latent). KV pool 518,167 tokens at MAXLEN=500000 with --gpu-memory-utilization 0.975 (the vision tower + projector add a fixed ~320 MiB/GPU the text-only build didn't, so the full 500K window needs 0.975, not 0.97). Override draft depth with -e NUM_SPEC=7 (structured/code) or -e NUM_SPEC=2 (general). Peel back any stack default with e.g. -e VLLM_MTP_INDEX_SHARE=0. The heavier NVFP4 weight per GPU leaves less KV room than the 1M variant; if a config OOMs, drop --max-model-len or --gpu-memory-utilization.

Validation

The vision graft here is identical to the 1M repo — same byte-identical MoonViT tower + projector, same glm5v code path — and was gated combined (vision + long context + MTP + LMCache, output byte-identical to the text-only reference) on the 1M build; see that card for the full gate. The text backbone is this variant's 48% NVFP4 hybrid, whose quality/throughput are the numbers above.

This specific quant level was then gated live end-to-end (2026-07-27): served the full 500K window (KV cache 518,167 tokens, TP4 + DCP4, KV fp8) and returned correct text (capital-of-France → "Paris") and correct image-color reads (solid-red → "red", solid-blue → "Blue") with reasoning (--reasoning-parser glm47) active. Vision tower and projector run in BF16 (tower sharded across TP ranks, projector replicated). Video input is untested.

Serving note: the vLLM build must be branch `vision-graft`, which carries the glm5v arch and the safetensors loader index-consistency fix (the re-tiered checkpoint leaves a stale pre-re-tier expert tensor in a base shard; stock vLLM's iterator over-reads it and crashes on a shape mismatch — the fix skips any tensor whose containing file isn't the one the index designates). This is a loader-only, lossless change; the weights in this repo are unchanged and correct.

code/ in the main repo has the full production pipeline (routing stats, assignment solver, checkpoint builders) and SETUP.md. Quantized from lukealonso/GLM-5.2-NVFP4.