GotoAI-Inc/Muse-Glimmer-30B-W4A16

Muse-Glimmer-30B-W4A16

Int4 weight-only quantization of meta-models/Muse-Glimmer-30B, in compressed-tensors format for vLLM. 22.20 GB, down from 59.55 GB — the model fits on a single 32 GB card, or a 24 GB card in text-only mode.

Unofficial and unaffiliated with Meta. All model capabilities, evaluations and limitations belong to the original model card — see the base model for those.

Meta publishes 4-bit builds of this model as llama.cpp K-quants and an ExecuTorch .pte, neither of which vLLM can load. This repository fills that gap: the same weights in the compressed-tensors format vLLM serves natively.

What was changed

Weights were quantized from bfloat16 to int4, group size 128, symmetric, weight-only (activations stay 16-bit) using llmcompressor.model_free_ptq. No calibration data was used and the model was never loaded — the quantizer operates directly on the safetensors. Architecture, tokenizer, chat template and processor config are the vendor's, unmodified.

The vision tower is deliberately untouched: vLLM builds Muse Glimmer's vision encoder with quant_config=None, so a checkpoint carrying quantized vision weights fails to load. lm_head is left at full precision as well — it is untied in this model, and it is the layer that decides the probabilities of the framing tokens that tool-call and reasoning parsing depend on.

Usage

Requires vLLM >= 0.28 (Muse Glimmer support landed after the 0.27.1 release) and transformers >= 5.15.

At the time of writing, 0.28 is not yet on PyPI, so vLLM must come from the nightly index. Note that nightly wheels self-report a lower version than the last release (0.26.1rc1.devNNNN+g<sha> vs 0.27.1), so pip install -U --pre vllm will silently keep 0.27.1 and the model will fail with invalid tool call parser: muse_glimmer. Pin the nightly explicitly:

pip install --force-reinstall --pre --extra-index-url https://wheels.vllm.ai/nightly \
  "vllm==0.26.1rc1.dev1059+g2740c817f"      # any main build after 2026-08-14 works
python -c "from vllm.model_executor.models.registry import ModelRegistry as R; \
print('MuseGlimmerForConditionalGeneration' in R.get_supported_archs())"

vllm serve GotoAI-Inc/Muse-Glimmer-30B-W4A16 \
  --max-model-len 65536 \
  --gpu-memory-utilization 0.90 \
  --enable-auto-tool-choice --tool-call-parser muse_glimmer \
  --reasoning-parser muse_glimmer

Do not pass --quantization; compressed-tensors is detected from config.json. The int4 W4A16 scheme uses Marlin kernels and runs on compute capability 7.5 and above.

• —`--tool-call-parser muse_glimmer` is not optional for agentic use: the model frames tool-call arguments in ATEM markup, and without the parser the named/required tool_choice path returns tool_calls: null.
• —`--reasoning-parser muse_glimmer` splits thinking into reasoning_content.
• —`--language-model-only` skips the vision tower and frees ~3.84 GB, which is what makes a 24 GB card viable — at the cost of image and video input.
• —Speculative decoding works through vLLM's DFlash path with the vendor's drafter: --speculative-config '{"method": "dflash", "model": "meta-models/Muse-Glimmer-30B-assistant", "num_speculative_tokens": 16}'

Fitting the card. --gpu-memory-utilization must cover weights and KV cache. 0.90 of a 32 GB card is comfortable; 0.90 of a 24 GB card is 21.6 GB, below the weights alone, so a 24 GB deployment needs --language-model-only. Long context is cheap here: 2 KV heads and a 2048 sliding window on 39 of 52 layers put a 32k sequence at roughly 0.5 GB, against a model maximum of 131072.

Reproducing this checkpoint

Built with llm-quantizer:

./llmq.py run --profile muse-glimmer-30b

which re-shards the source (it ships as a single 49.95 GB safetensors file, which no consumer GPU can hold) into 4 GB pieces, then:

# llmcompressor==0.13.1a20260814, compressed-tensors==0.18.1a20260818,
# transformers==5.15.1, torch==2.13.0
from llmcompressor import model_free_ptq

model_free_ptq(
    model_stub="Muse-Glimmer-30B-resharded",
    save_directory="Muse-Glimmer-30B-W4A16",
    scheme="W4A16",
    ignore=["re:.*vision.*", "lm_head", "re:.*embed_tokens.*"],
    device="cuda:0",
)

Because a job holds one shard at a time, the peak device footprint is a few GB rather than the model size — this build ran on a 16 GB GPU.

Evaluation

No benchmarks have been run. Treat the published Muse Glimmer numbers as describing the bfloat16 model, not this one: data-free round-to-nearest quantization degrades quality more than a calibrated (GPTQ/AWQ) or QAT build, and how much, for your task, is unmeasured here.

What has been checked is that it works in practice. The checkpoint has been served on a single 48 GB GPU under vLLM and used to drive the Goose agent harness through normal sessions, with --tool-call-parser muse_glimmer and --reasoning-parser muse_glimmer. Multi-step tool calling and reasoning extraction behave as expected — no malformed tool calls or leaked framing tokens observed.

That is a functional smoke test, not a quality measurement, but it exercises the part most sensitive to weight quantization: structured emission (tool-call markup and reasoning delimiters) degrades before fluency does, so an agent harness running cleanly is more informative here than a perplexity number would be.

License

Apache 2.0, inherited from the base model — the vendor's LICENSE is included. The base model's `USAGE_POLICY.md` is also included and applies to this derivative; please read it before use. "Muse Glimmer" is Meta's; this repository is not endorsed by or affiliated with Meta.