Kaleto/ZGCM-1-7B-NVFP4

ZGCM-1-7B — NVFP4 (W4A16)

NVFP4 (4-bit floating-point weights, group_size=16) quantization of zgcagi/ZGCM-1-7B, a 7.39B dense model trained from scratch for math reasoning and agentic search (256K context, thinking and direct-response modes).

No official quantizations existed at the time of release. Built and tested on an NVIDIA DGX Spark (GB10), where it decodes 1.5× faster than online FP8.

The checkpoint is marked quant_algo: W4A16_NVFP4: vLLM runs it weight-only with the Marlin kernel (4-bit weights, BF16 activations). See Why W4A16 before changing that.

Quick facts

Benchmarks (DGX Spark, single GB10)

vLLM 0.26.0, --model-impl transformers --enforce-eager, FP8 KV cache, batch size 1, streaming decode speed. Thinking mode, temperature 0.6, top_p 0.95, 350–400 generated tokens per prompt, 3 prompts (arithmetic, Python, short explanation).

Decode on GB10 is memory-bandwidth bound: at 34.5 tok/s the model reads ~6 GB per token. Of that, the BF16 lm_head (0.64B parameters, 1.27 GB) is now roughly a fifth.

Quality: smoke-tested with coherent thinking traces and correct answers on arithmetic, code and explanation prompts. On multi-turn few-shot prompts in non-thinking mode, the model can degenerate into !!!!. The cause is in the base model; see Known issue. A KL-divergence comparison against BF16 is in KL divergence; it is confounded by the same base-model issue. Treat math/benchmark scores from the base card as unverified for this quantization.

KL divergence vs BF16

Measured on 25 texts (English and Chinese Wikipedia, MATH-500, HumanEval, MMLU-Pro 5-shot chat prompts; ~39k positions), from vLLM prompt log-probabilities (top-50 plus a rest bucket, which gives a lower bound of the full-vocabulary KLD).

The numbers are hard to interpret for this model. The BF16 base model itself predicts `!` as the top token at 40–99 % of the positions of ordinary text (with its [gMASK]<sop> prefix; see Known issue). Both models agree on that most of the time, which pushes the median KLD towards zero.

On the only texts where the base model stays out of that mode, NVFP4 is clearly lossier than NVFP4 on large models: DeepSeek-R1-Distill-70B measured ~0.06 with full-vocabulary KLD. This is uncalibrated round-to-nearest quantization of a 7B model with very large activations.

Known issue: !!!! degeneration on multi-turn few-shot prompts

What happens. In non-thinking mode (enable_thinking: false), greedy decoding, with the prompt given as a multi-turn few-shot dialogue (e.g. MMLU-Pro CoT: system prompt + 5 question/answer turns), some answers collapse into ! (token id 0). The collapse either happens at the first token or starts as isolated ! between words (We! refer! to! …) before turning into a !!!! loop until the token limit. It depends strongly on the content of the few-shot block. In an MMLU-Pro run (300 questions), all 28 chemistry questions collapsed and none of the 28 law questions did. Overall 135 of 2102 answers across 9 tasks ended in a loop. 115 of them were in the two MMLU-Pro-style multi-turn tasks (English and German), 19 in short German single-turn prompts (INCLUDE), where the model already writes broken German before collapsing. The analysis below covers the multi-turn case; the German single-turn cases were not compared against BF16.

The cause is in the original model, not in this quantization or in vLLM. The same prompts were checked outside vLLM with the HF remote code on CPU: NVFP4 weights dequantized exactly, no FP8 KV cache, no Marlin kernel, no prefix cache. The BF16 weights of zgcagi/ZGCM-1-7B were run under identical conditions. Top-1 next token (log-probability):

• —The logits are finite (no NaN/Inf): ! is a genuine, confident prediction of the base model, not a numerical failure.
• —Confirmed on GPU with the BF16 release and plain transformers (AutoModelForCausalLM, remote code, greedy generate, no vLLM): P(first token = !) is 0.995 for chemistry #7 and 0.988 for chemistry #83. Reported upstream as zgcagi/ZGCM-1#5.
• —The trigger is the few-shot history given as chat turns. Asked zero-shot (system prompt + the same question only), or with the same 5 examples inside a single user message, both chemistry questions start normally in BF16 (P(!) = 0.000).
• —Enabling thinking does not avoid it on these few-shot prompts. With <think> opened, ! is still the top token (P = 0.90 and 0.97 in BF16).
• —It is not limited to few-shot prompts. Used as a plain text-completion model (BF16, plain transformers, with the [gMASK]<sop> prefix), the base model predicts ! as the top token at 99 % of positions of a Wikipedia article and of MATH-500 text, and at 55 % in HumanEval code. Greedy continuation after 200 tokens of Wikipedia is !!!!…. Use the chat template with the assistant turn; do not use this model for raw text completion.
• —The remote code matches the published Megatron training code (RMSNorm, pre-LN, QK-norm, partial RoPE, gated sliding-window attention, window size). The prompt format matches the SFT format. The base model also has very large trained activations in its last layer (residual values of 1e8–1e11 in BF16).

The quantization does not make it worse. Both models were run on the same 300 MMLU-Pro prompts (5-shot chat, no-think, greedy, max 512 tokens), with the same vLLM 0.26 settings on the same GB10:

122 prompts collapse with both, 73 only with BF16, 7 only with NVFP4. Individual prompts can go either way (history #152 above collapses only with NVFP4). vLLM is not fully deterministic here: two NVFP4 passes differ on about 30 prompts, so the 66-prompt gap is real, but single-prompt differences are not.

Recommendations

• —Put few-shot examples inside one user message rather than as separate chat turns (tested: both chemistry prompts then answer normally).
• —Thinking mode does not help on multi-turn few-shot prompts. For single-turn reasoning, use it with the sampling settings of the base card.
• —For benchmark harnesses that send multi-turn few-shot prompts in non-thinking mode (e.g. lm-eval MMLU-Pro chat), expect !!!! answers. The BF16 base model degenerates even more often. Report them as a property of ZGCM-1, not of this quantization.

Why W4A16 and not W4A4

The same weights can be served two ways, selected only by the quant_algo string in config.json:

With NVFP4, vLLM also quantizes activations to FP4. This checkpoint carries no calibrated activation scales (input_scale = 1.0), which this model does not tolerate. W4A4 was slower on GB10 anyway. Keep W4A16_NVFP4.

Usage

vLLM (serve)

vllm serve Kaleto/ZGCM-1-7B-NVFP4 \
  --trust-remote-code \
  --model-impl transformers \
  --enforce-eager \
  --reasoning-parser glm45 \
  --enable-auto-tool-choice \
  --tool-call-parser glm47 \
  --kv-cache-dtype fp8 \
  --max-model-len 262144 \
  --max-num-seqs 6 \
  --gpu-memory-utilization 0.28 \
  --enable-chunked-prefill \
  --max-num-batched-tokens 8192 \
  --enable-prefix-caching

• —No --quantization flag needed; vLLM detects the modelopt config (Detected ModelOpt NVFP4 checkpoint (quant_algo=W4A16_NVFP4)).
• —`--enforce-eager` is required. With torch.compile / CUDA graphs, output collapses into a backtick loop after ~30 tokens. The same collapse occurs with the online-FP8 variant (with FP8 and BF16 KV cache alike), so it is not specific to this quantization; eager mode is clean.
• —--model-impl transformers is required: vLLM has no native ZGCM implementation. The remote code dispatches attention to vLLM itself.
• —The chat template uses the GLM-4.5/4.7 format, hence the glm45 / glm47 parsers.
• —KV cache is small: only the 5 global-attention layers keep the full context, the 27 sliding-window layers only their 128-token window. At --gpu-memory-utilization 0.22 with 32K max length, vLLM allocated 15.98 GiB KV = 416,117 tokens.
• —--gpu-memory-utilization is a fraction of total memory (128 GB on a Spark); scale it to your GPU.

Sampling

The base card evaluates with temperature 1.0 and topp 1.0 in thinking mode. For interactive use, temperature 0.6 / topp 0.95 worked well in testing. Tool calls are much more reliable with thinking enabled.

Post-export fixes

modelopt 0.43's HF export needs adjustments for vLLM, all applied to this checkpoint:

1. 1.input_scale = 1.0 injected for every quantized linear (251×; modelopt omits them for weight-only export)
2. 2.config_groups.*.input_activations.dynamic = true
3. 3.quantization_config.ignore = ["lm_head"]
4. 4.quant_algo set to W4A16_NVFP4 in config.json and hf_quant_config.json
5. 5.configuration_zgcm.py, modeling_zgcm.py, auto_map, tokenizer and chat_template.jinja copied unchanged from the base model

Conversion ran on a single GB10 with modelopt 0.43 and transformers 5.8 (mtq.quantize with NVFP4_DEFAULT_CFG, *lm_head* / *embed_tokens* / *norm* disabled, then export_hf_checkpoint).

Files

• —model-00001-of-00002.safetensors, model-00002-of-00002.safetensors — NVFP4 weights + scales, BF16 embeddings / lm_head / norms
• —model.safetensors.index.json
• —config.json — upstream config + quantization_config (modelopt, W4A16_NVFP4)
• —hf_quant_config.json, generation_config.json
• —configuration_zgcm.py, modeling_zgcm.py — unchanged from the base model
• —tokenizer.json, tokenizer_config.json, chat_template.jinja — unchanged from the base model
• —LICENSE — MIT, from the base model

Acknowledgments

• —[zgcagi](https://huggingface.co/zgcagi) (Zhongguancun Academy / Zhongguancun Institute of Artificial Intelligence) for ZGCM-1 — see the tech report and training code
• —NVIDIA for DGX Spark / GB10, NVFP4 and modelopt
• —vLLM project for modelopt NVFP4 support and the transformers backend

License

MIT, inherited from zgcagi/ZGCM-1-7B.

Citation

Please cite the original work:

@misc{zgcm1,
  title={ZGCM-1: A Fully Open and Extremely Efficient Foundation Model for Math and Agentic Search},
  author={Jiyan He and Guang Liang and Hao Liu and Haoxiang Guan and Jinbo Sun and Junyi Guo and Wenjun Feng and Yantai Xie and Yifei Shen and Bin Shao and Chuyang Wei and Kai Chen and Kexin Zhou and Minghang Zhu and Shuxin Zheng and Tie-Yan Liu and Taine Zhao and Wenhui Zhu and Xueyin Xu and Xiaoqing Zhang and Yatao Li and Yuxuan Ren},
  year={2026},
  eprint={2609.13356},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/abs/2609.13356}
}