vroomfondel/glm-5.2-reap-504B-v2-W4A4

GLM-5.2-504B-REAP-v2 — attention projections quantized to NVFP4 W4A4

⚠️ EXPERIMENTAL RESEARCH ARTIFACT — a one-off DGX Spark test run, not a production release. Produced specifically for NVIDIA GB10 / SM121a (DGX Spark) serving, to measure whether NVFP4-quantizing the MLA attention projections recovers decode throughput on consumer Blackwell. The input_scale is a heuristic (no calibration), so correctness was not guaranteed a priori — but it has since been empirically validated: GSM8K 93.0% (0 errors) and the intended single-stream decode speedup confirmed (see Validation results below). Still treat it as a research artifact: the heuristic scale simply held for this checkpoint (not a calibrated release), and this is a single-checkpoint, GB10/SM121a-specific experiment — but accuracy, decode speedup and loop/termination behavior have now all been measured (see Validation results).

A derivative of `0xSero/glm-5.2-reap-504B-v2` that additionally quantizes the two heavy MLA attention projections (o_proj, q_b_proj) from BF16 to NVFP4 W4A4, cutting single-stream decode weight-bandwidth on the attention path. Everything else is byte-identical to the base model. NOT an 0xSero release — this is a downstream serving-optimization experiment produced on and for the dgxarley DGX Spark cluster (4× GB10 / SM121a, ARM64, TP4).

📄 Full technical report — motivation (decode profiling), method, the two NVFP4 ignore-list gotchas, benchmarks and validation: `reap_requant.md`. The exact serving config is the SGLang model profile `vroomfondel-glm-5.2-reap-504b-v2-w4a4.yml`.

What this variant changes (and why)

On a 4× DGX Spark (GB10 / SM121, consumer Blackwell, TP4) live decode profile, the unquantized BF16 MLA-projection GEMVs were ~71 of ~131 ms per decode token — pure weight bandwidth at batch size 1. o_proj (6144×16384) and q_b_proj (16384×2048) carry ~80% of the attention weight bytes. Quantizing just those two to NVFP4 removes most of that floor.

How it was produced (data-free, no re-calibration)

• —Weights (weight, weight_scale, weight_scale_2): standard modelopt NVFP4 packing (e2m1 fp4, per-16 e4m3 block scales, per-tensor fp32 global), computed data-free from the BF16 weights. The exact packing/scale layout (linear block scales, weight_scale_2 = amax / (6·448)) was verified against an existing NVFP4 expert tensor in the base checkpoint (dequant round-trip ~0.09 rel-err, i.e. ordinary NVFP4 fidelity).
• —`input_scale` (static, required by the W4A4 dense-linear path — dynamic activation quant is not supported by that kernel): a heuristic generous per-tensor estimate — no calibration pass was run (the 504B model does not fit single-node PTQ). This is the one accuracy-relevant approximation; the GSM8K gate below is what validates it — and it PASSED (93.0%, 0 errors), so the heuristic holds for this checkpoint.
• —Config: o_proj/q_b_proj are dropped from quantization_config.ignore so they fall into the existing W4A4 group (same algo as the experts); no MIXED_PRECISION needed.

Kernel choice: cutlass W4A4 (not Marlin W4A16)

Measured on GB10 across a batch sweep: Marlin (the W4A16 weight-only path) collapses above ~bs 64 and is unstable at bs 1; the cutlass NVFP4 W4A4 path is stable across batch sizes and decisively faster at high concurrency. This variant therefore serves via the cutlass scaled_fp4_mm path (both weights AND activations NVFP4).

Validation results (dgxarley cluster, 2026-07-17)

Deployed on 4× GB10 / SM121a (TP4, DSA sparse attention, MTP/NEXTN speculative decoding on) and gated:

• —GSM8K accuracy — PASS. 5-shot, greedy (temp 0), n=200: 93.0% flexible AND strict (186/200), 0 errors / 0 empty responses. A broken W4A4 (bad input_scale) would score <70% or emit garbage — 93% with zero failures means the heuristic input_scale (no calibration) holds; accuracy is not degraded. (The base's own GSM8K figure was measured with a different harness/shot count, so this is a standalone go/no-go, not a paired A/B on identical items.)
• —Decode throughput — higher, as intended. Single-stream (batch 1) with MTP: measured ~18–29 tok/s (accept len ~3–4 depending on prompt predictability), vs the un-requantized base's ~11.7–12.4 tok/s reference (accept ~2.1). Normalizing out MTP acceptance (tok/s ÷ accept-len = raw target forward-pass rate), the requant-attributable speedup is ~15–18% per forward pass — the o_proj/q_b_proj NVFP4 GEMV win. The larger end-to-end delta partly reflects higher MTP acceptance on the test prompts (the base was not re-measured on identical prompts), so treat the ~15–18% as the clean requant-attributable number.
• —Loader note (config correctness). SGLang's modelopt loader fuses q_a_proj+kv_a_proj_with_mqa into a single runtime param fused_qkv_a_proj_with_mqa and matches ignore entries against that fused name by exact last-segment set-membership. So the surgical ignore list must (a) emit the fused-component entries *without a trailing `**, and (b) keep the un-requantized **MTP layer's** oproj`/`qb_proj in ignore (BF16). Both are corrected in the shipped config.json`; without them the checkpoint fails to load with a W4A4-vs-BF16 shape assertion.
• —Loop/attractor behavior — no regression. Measured with the cluster's streaming repetition-detector over 250 open-ended prompts × 3 sampling conditions (0 errors / 750 generations), monitoring both the answer and the thinking stream: raw (no guardrails) 2.8%, and the base's recommended_sampling keeps it near-zero — min_p=0.05 + repetition_penalty=1.05 → 0.4%, 1.10 → 0.8%. Every rare loop was in the thinking stream, none in the final answer → NVFP4- rounding the KD-LoRA-bearing o_proj/q_b_proj did not worsen termination. (Our harness / prompt set / loop definition, not the base's exact n=2000 agent-prompt methodology, so not a paired A/B against its documented 7.2% / 4.9% / 2.3%.)

⚠️ Still an experimental research artifact — heuristic input_scale, a single checkpoint, GB10/SM121a-specific — but accuracy, decode throughput and loop behavior have now all been measured (above).

Serving

Same architecture and launch as the base model (--quantization modelopt_fp4, --trust-remote-code). On SM120/SM121 (consumer Blackwell) the DSA sparse attention path additionally needs the dgxarley DSA runtime patches (native SM120 sparse-MLA routing + the torch/Triton indexer fallback; see the upstream sglang PRs). This variant is intended for that stack.

The exact SGLang launch used on the cluster (head node of a 4-node TP4 setup; --node-rank/--nccl-init-addr/--host adjust per node/network):

python3 -m sglang.launch_server \
  --model-path vroomfondel/glm-5.2-reap-504B-v2-W4A4 \
  --served-model-name vroomfondel/glm-5.2-reap-504B-v2-W4A4 \
  --quantization modelopt_fp4 --trust-remote-code \
  --tp-size 4 --pp-size 1 --nnodes 4 --node-rank 0 \
  --nccl-init-addr 10.10.100.1:50000 --host 127.0.0.1 --port 30080 \
  --context-length 16384 --kv-cache-dtype fp8_e4m3 --mem-fraction-static 0.99 \
  --max-running-requests 32 --max-total-tokens 131072 --schedule-policy lpm \
  --attention-backend dsa --dsa-decode-backend trtllm --dsa-prefill-backend trtllm \
  --dsa-paged-mqa-logits-backend torch \
  --moe-runner-backend flashinfer_cutlass --fp4-gemm-backend flashinfer_cutlass \
  --speculative-algo NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --reasoning-parser glm45 --tool-call-parser glm47 --enable-custom-logit-processor \
  --cuda-graph-max-bs 32 --disable-piecewise-cuda-graph \
  --dist-timeout 1800 --watchdog-timeout 3600 --enable-metrics

Everything below is inherited from the base model (0xSero's work) — preserved for completeness

The base model 0xSero/glm-5.2-reap-504B-v2 is a 34%-expert-pruned GLM-5.2 (GlmMoeDsaForCausalLM, 78 layers + 1 MTP, DeepSeek-style MLA + DSA sparse indexer, hidden 6144), recovered to near-parity by training only the router gates. All of the following is 0xSero's methodology and measurement, not ours.

• —Prune method: REAP — saliency = gate_weight × ‖expert_output‖, top-168 of 256 experts kept per layer (consistently across MoE + MTP layers → loads cleanly).
• —Recovery: Router-KD — freeze the whole network, train ONLY the 75 router gate matrices (~0.016% of params) to KL-match the unpruned GLM-5.2 teacher.
• —Base quantization: NVFP4 (modelopt) on the routed experts; BF16 router / attention / shared expert. (This variant extends NVFP4 to o_proj/q_b_proj.)
• —Honest cost (n=2000 held-out agent prompts, raw sampling): loop/attractor rate 7.2% vs the unpruned teacher's 3.6% — pruning + gate-only Router-KD roughly doubles looping (z≈5, p<0.0001); Router-KD recovers routing but not the termination behavior carried by the pruned experts.
• —Recommended sampling (recovers the loop gap for free): min_p=0.05, repetition_penalty=1.05 (→ 4.9% loops) up to 1.10 (→ 2.3%, fully recovers). Optional brevity system prompt halves median length.
• —Lineage / reproducibility: base = zai-org/GLM-5.2; the REAP cut + KD gates
• —logit-KD LoRA are reconstructable from 0xSero/GLM-5.2-504B-REAP-recovery-kit.
• —Full technical report: see the base model's REPORT.md.
• —Compute for the base model was sponsored by Lambda (8× B200). 🙏

(--dist-timeout 1800 + a generous liveness grace matter on the slow-cold-load JuiceFS setup; see the technical report.)

Derivative of 0xSero's REAP + Router-KD + NVFP4 work; only the `o_proj`/`q_b_proj` NVFP4 W4A4 quantization and this card are ours. Not affiliated with 0xSero.