vroomfondel/GLM-5.3-Flash-NVFP4-W4A4
GLM-5.3-Flash-NVFP4-W4A4
A surgical NVFP4 top-up of LibertAIDAI/GLM-5.3-Flash-NVFP4, which is itself an NVIDIA ModelOpt NVFP4 quantization of zai-org/GLM-5.3-Flash.
Not quality-validated. No token has been generated from this checkpoint. The glm5_next architecture is not in any released SGLang or vLLM at the time of writing, so the structural and numeric checks below are all that back it. Treat it as a research artifact, not a drop-in replacement for the base checkpoint.What this changes
The base checkpoint quantizes the routed experts and nothing else. That is about 95% of the parameters, but at batch size 1 only 8 of 288 experts are read per token while the whole attention stack is read every token, so the remaining BF16 weights dominate the per-token weight traffic. Rough estimate: routed-expert active weights are ~8.5B params at ~0.5625 bytes (~4.8 GB per decode token) against ~8.3B params of BF16 at 2 bytes (~16.6 GB).
This build quantizes the always-read remainder that is safe to touch:
191 tensors, 3.67B parameters. Already-quantized expert tensors are passed through byte-identically, so nothing is quantized twice and no rounding error is compounded.
Every quantized linear, old and new, is NVFP4 W4A4: 4-bit E2M1 weights in groups of 16 with FP8 block scales plus a per-tensor weight_scale_2, and statically scaled 4-bit activations via input_scale. The "input_activations": null inherited in config.json is misleading; quant_algo: NVFP4 together with the input_scale tensors is what selects the W4A4 path in the modelopt loaders. This repo was previously named GLM-5.3-Flash-NVFP4-ToppedUp.
What deliberately stays BF16
The KDA linear-attention state path (q/k/v_conv1d, f_a/f_b_proj, g_a/g_b_proj, b_proj, o_norm, A_log, dt_bias), the MLA low-rank compressions (q_a_proj, kv_a_proj_with_mqa, kv_b_proj), the DSA indexer, the routers, the mHC hyper-connection parameters, the MTP block (layer 45), the vision tower, lm_head and embed_tokens.
The KDA q/k/v_proj are also left alone, deliberately. SGLang fuses six projections of each linear-attention layer into two GEMMs and only does so while they are unquantized (sglang#37744); five of the six are dense holds here for correctness, so quantizing the remaining three would trade that fusion away for a modest bandwidth gain. Whether that is a net win is unmeasured.
Verification
NVFP4 round-trip mean relative error, dequantized against the source BF16: o_proj 0.0902, q_b_proj 0.0878, shared_experts.down_proj 0.0897, dense gate_proj 0.0875. Consistent across shapes and module families, which is the point of the check.
Known limitation: activation scales are heuristic
The NVFP4 weights are data-free and identical to what a calibrated run would produce. The per-tensor input_scale is not calibrated: it is a deliberately generous amax estimate (16.0), which over-estimates rather than clips. As a sanity check, this yields 0.005952 against the base checkpoint's calibrated expert scale of 0.005376, i.e. 11% more generous, in the intended direction. Expert activations are not attention activations, so that is indicative and not proof.
Quantizing from full precision with real calibration is the only way to do better here, and it needs the 642 GB BF16 source plus a multi-node run.
Provenance
Produced on a single NVIDIA DGX Spark (GB10 / sm121) with flashinfer 0.6.17, streaming shard by shard without ever loading the model, by surgical_glm5next_nvfp4.py from the KIKUBE quantizer toolchain.
python3 surgical_glm5next_nvfp4.py \
--src <LibertAIDAI/GLM-5.3-Flash-NVFP4 snapshot> \
--out <hf cache hub dir> \
--repo-id vroomfondel/GLM-5.3-Flash-NVFP4-W4A4Inherits the MIT license of the base model.