abstract-extraordinary/Muse-Glimmer-30B-NVFP4

Muse-Glimmer-30B-NVFP4

NVFP4 (4-bit weights, 4-bit activations) quantization of meta-models/Muse-Glimmer-30B, made with llm-compressor. 55.5 GiB to 21.8 GiB.

Quantized and tested on a single NVIDIA GB10 (DGX Spark, sm_121, 121 GiB unified memory).

Requirements

Muse Glimmer is not in any released vLLM. Support comes from PR #51655, which is still open. As of 2026-08-11 that PR is enough to serve this checkpoint.

Usage

vllm serve abstract-extraordinary/Muse-Glimmer-30B-NVFP4 \
  --max-model-len 131072 \
  --max-num-seqs 4 \
  --gpu-memory-utilization 0.45 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --compilation-config '{"cudagraph_mode":"PIECEWISE"}' \
  --reasoning-parser muse_glimmer \
  --tool-call-parser muse_glimmer \
  --enable-auto-tool-choice

Both parsers are required, not optional. The model emits ATEM XML tool calls on channel-scoped messages and both parsers key off that framing. On GB10, cudagraph_mode: PIECEWISE avoids a silent hang. Multi-modal warmup failed at startup is harmless; image requests work.

What is quantized

416 Linear modules in the text decoder: 52 layers of q,k,v,o,gate plus mlp.gate,up,down. Weights are 4-bit float at group size 16 with FP8 group scales and an FP32 per-tensor global scale; activations are 4-bit float, quantized dynamically per group at runtime. Format is nvfp4-pack-quantized (compressed-tensors 0.15.1).

The vision tower is bf16 because vLLM's Muse Glimmer implementation constructs every vision linear without a quant_config, so a quantized ViT cannot be loaded at all. It costs about 3.6 GiB. lm_head is bf16 by convention.

Evaluation

GSM8K, first 250 test questions, zero-shot with an explicit #### <number> answer format, greedy, exact match. Both checkpoints were served by the same vLLM build with identical settings and scored by the same harness.

The two-point gap is not significant and should not be read as an improvement. Paired over the same questions: 234 correct under both, 9 wrong under both, 6 correct only under NVFP4, 1 only under bf16, McNemar exact p = 0.125. Greedy decoding in vLLM is also not bit-reproducible between runs. The result supports no measurable quantization loss, and nothing stronger.

Five NVFP4 and six bf16 generations hit the 2048-token cap and score as wrong, which pulls both numbers down about equally. No request errors on either side. At n=250 the eval resolves differences of roughly three points.

Text, image input and tool calling were each checked after quantization and behave as before.

Performance

Single GB10, TP=1, no speculative decoding. Both columns come from the GSM8K runs above.

At 131k context the KV cache is 29.3 GiB (1,645,658 tokens) for 12.56x maximum concurrency, with about 62 GiB resident against roughly 90 GiB for bf16.

Single-stream decode measures 12.7 tok/s. There is no paired bf16 single-stream figure, so use the batch numbers above for comparison.

Method

QuantizationModifier(
    targets="Linear",
    scheme="NVFP4",
    ignore=["re:.*lm_head", "re:.*vision_tower.*",
            "re:.*vision_adapter.*", "re:.*vision_projection.*"],
)

Calibration used 32 samples from neuralmagic/calibration (LLM split) at 1024 tokens, rendered through the model's own chat template so the channel framing is present. Text only, since the vision tower is excluded.

llm-compressor traced the model into 53 sequential subgraphs, the 52 decoder layers plus one. The 50 vision layers do not trace under text-only calibration, which is expected and harmless given they are excluded anyway.

Building the model requires transformers from main. muse_glimmer is in no released version and the base repo ships no remote code.

Limitations

• —Requires a vLLM build carrying PR #51655, which is still unmerged.
• —Vision output was checked qualitatively but not benchmarked. The vision tower is bf16 and bit-identical to the original; only the decoder it feeds changed.
• —Speculative decoding with the DFlash drafter is untested against this target. The drafter was trained against the bf16 model.
• —Accuracy comes from one task and 250 samples. It says nothing about long-context, multilingual or coding behaviour.

License

Apache 2.0, inherited from the base model, whose LICENSE and USAGE_POLICY.md are included here and apply unchanged. All credit for the model belongs to its original authors; this repository holds a quantized copy of their weights.