GotoAI-Inc/Muse-Glimmer-30B-W8A16

Muse-Glimmer-30B-W8A16

Int8 weight-only quantization of meta-models/Muse-Glimmer-30B, in compressed-tensors format for vLLM. 34.78 GB, down from 59.55 GB — the model fits a 48 GB card, or a 40 GB card comfortably (its KV cache is tiny); text-only it is 30.94 GB.

This is the fidelity-first build. Int8 round-to-nearest stays far closer to the bfloat16 weights than int4 does, at ~1.6x the footprint of the int4 W4A16 sibling (22.20 GB). If you are targeting a 24 GB card, use that one; use this one when you have the VRAM and want the least quality loss quantization can give without calibration.

Unofficial and unaffiliated with Meta. All model capabilities, evaluations and limitations belong to the original model card — see the base model for those.

Meta publishes 4-bit builds of this model as llama.cpp K-quants and an ExecuTorch .pte, neither of which vLLM can load. This repository fills that gap: the same weights in the compressed-tensors format vLLM serves natively.

What was changed

Weights were quantized from bfloat16 to int8, group size 128, symmetric, weight-only (activations stay 16-bit) using llmcompressor.model_free_ptq. No calibration data was used and the model was never loaded — the quantizer operates directly on the safetensors. Architecture, tokenizer, chat template and processor config are the vendor's, unmodified.

The vision tower is deliberately untouched: vLLM builds Muse Glimmer's vision encoder with quant_config=None, so a checkpoint carrying quantized vision weights fails to load. lm_head is left at full precision as well — it is untied in this model, and it is the layer that decides the probabilities of the framing tokens that tool-call and reasoning parsing depend on.

Usage

Requires vLLM >= 0.28 (Muse Glimmer support landed after the 0.27.1 release) and transformers >= 5.15.

At the time of writing, 0.28 is not yet on PyPI, so vLLM must come from the nightly index. Note that nightly wheels self-report a lower version than the last release (0.26.1rc1.devNNNN+g<sha> vs 0.27.1), so pip install -U --pre vllm will silently keep 0.27.1 and the model will fail with invalid tool call parser: muse_glimmer. Pin the nightly explicitly:

pip install --force-reinstall --pre --extra-index-url https://wheels.vllm.ai/nightly \
  "vllm==0.26.1rc1.dev1059+g2740c817f"      # any main build after 2026-08-14 works
python -c "from vllm.model_executor.models.registry import ModelRegistry as R; \
print('MuseGlimmerForConditionalGeneration' in R.get_supported_archs())"

vllm serve GotoAI-Inc/Muse-Glimmer-30B-W8A16 \
  --max-model-len 65536 \
  --gpu-memory-utilization 0.90 \
  --enable-auto-tool-choice --tool-call-parser muse_glimmer \
  --reasoning-parser muse_glimmer

Do not pass --quantization; compressed-tensors is detected from config.json. The int8 W8A16 scheme uses Marlin kernels and runs on compute capability 7.5 and above.

• —`--tool-call-parser muse_glimmer` is not optional for agentic use: the model frames tool-call arguments in ATEM markup, and without the parser the named/required tool_choice path returns tool_calls: null.
• —`--reasoning-parser muse_glimmer` splits thinking into reasoning_content.
• —`--language-model-only` skips the vision tower and frees ~3.84 GB (a 30.94 GB text-only checkpoint), at the cost of image and video input.
• —Speculative decoding works through vLLM's DFlash path with the vendor's drafter: --speculative-config '{"method": "dflash", "model": "meta-models/Muse-Glimmer-30B-assistant", "num_speculative_tokens": 16}'

Fitting the card. --gpu-memory-utilization must cover weights and KV cache. 0.90 of a 48 GB card is 43.2 GB — comfortable over the 34.78 GB of weights. 0.90 of a 40 GB card is 36 GB, which still clears the weights because long context is cheap here: 2 KV heads and a 2048 sliding window on 39 of 52 layers put a 32k sequence at roughly 0.5 GB, against a model maximum of 131072. On a 32 GB card use --language-model-only to drop to the 30.94 GB text-only footprint.

Reproducing this checkpoint

Built with llm-quantizer:

./llmq.py run --profile muse-glimmer-30b

W8A16 is the profile's default scheme, so no --scheme flag is needed. The run re-shards the source (it ships as a single 49.95 GB safetensors file, which no consumer GPU can hold) into 4 GB pieces, then:

# llmcompressor==0.13.1a20260814, compressed-tensors==0.18.1a20260818,
# transformers==5.15.1, torch==2.13.0
from llmcompressor import model_free_ptq

model_free_ptq(
    model_stub="Muse-Glimmer-30B-resharded",
    save_directory="Muse-Glimmer-30B-W8A16",
    scheme="W8A16",
    ignore=["re:.*vision.*", "lm_head", "re:.*embed_tokens.*"],
    device="cuda:0",
)

Because a job holds one shard at a time, the peak device footprint is a few GB rather than the model size — this build ran on a 16 GB GPU.

Evaluation

No benchmarks have been run. Treat the published Muse Glimmer numbers as describing the bfloat16 model, not this one: data-free round-to-nearest quantization degrades quality more than a calibrated (GPTQ/AWQ) or QAT build, and how much, for your task, is unmeasured here. Int8 degrades far less than the int4 sibling — that is the reason this build exists — but "less" is not "none".

What has been checked is that it works in practice. The checkpoint has been served on a single 48 GB GPU under vLLM and used to drive the Goose agent harness through normal sessions, with --tool-call-parser muse_glimmer and --reasoning-parser muse_glimmer. Multi-step tool calling and reasoning extraction behave as expected — no malformed tool calls or leaked framing tokens observed.

That is a functional smoke test, not a quality measurement, but it exercises the part most sensitive to weight quantization: structured emission (tool-call markup and reasoning delimiters) degrades before fluency does, so an agent harness running cleanly is more informative here than a perplexity number would be.

License

Apache 2.0, inherited from the base model — the vendor's LICENSE is included. The base model's `USAGE_POLICY.md` is also included and applies to this derivative; please read it before use. "Muse Glimmer" is Meta's; this repository is not endorsed by or affiliated with Meta.