TheHouseOfTheDude/Muse-Glimmer-30B_INT8-PTQ

Muse-Glimmer-30B-INT8 — PTQ Quantized (W8A16)

Overview

Post-Training Quantized (PTQ) version of meta-models/Muse-Glimmer-30B.

Muse Glimmer-30B is a dense multimodal (vision-language) agentic model (~29.6B total incl. vision) with a Perception Encoder (ViT-G/14, ~1.8B), hybrid sliding-window / full attention (3:1), and 131K context. This checkpoint keeps the vision / perception path in BF16 and quantizes the language-model Linear layers to INT8.

• —No calibration dataset
• —One-shot quantization (llmcompressor.oneshot)
• —Uses llmcompressor QuantizationModifier with the W8A16 preset (no AWQ, no GPTQ)
• —Loaded / saved via MuseGlimmerForConditionalGeneration so vLLM sees the correct language_model / vision_* weight paths
• —Compressed with save_pretrained(..., save_compressed=True) (compressed-tensors)
• —Processor / tokenizer / chat-template sidecars copied from the BF16 source for a complete multimodal package
• —Source generation_config.json restored so list-valued eos_token_id is preserved for multi-stop decoding in vLLM

Quantization

• —Scheme: W8A16
• —Weights: INT8 (per-channel, symmetric)
• —Activations: FP16/BF16 (untouched)
• —Targets: Linear layers only
• —Modifier: QuantizationModifier(targets="Linear", scheme="W8A16", ...)

Ignored (left unquantized)

Dense model: no MoE router / expert ignores.

Recipe (from the quant script)

from llmcompressor.modifiers.quantization import QuantizationModifier

recipe = QuantizationModifier(
    targets="Linear",
    scheme="W8A16",
    ignore=[
        "lm_head",
        "re:.*vision_tower.*",
        "re:.*vision_adapter.*",
        "re:.*vision_projection.*",
        "re:.*multi_modal_projector.*",
        "re:.*mm_projector.*",
        "re:.*draft.*",
        "re:.*dflash.*",
    ],
)

Usage

vllm serve TheHouseOfTheDude/Muse-Glimmer-30B-INT8 \
  --quantization compressed-tensors

Tensor-parallel example:

vllm serve TheHouseOfTheDude/Muse-Glimmer-30B-INT8 \
  --quantization compressed-tensors \
  -tp <num_gpus>

Recommended sampling (from base model)

• —temperature = 1.0
• —top_p = 0.95
• —top_k = 64

Reasoning strength can be set in the system prompt as Reasoning strength: <low|medium|high|xhigh>.

Notes

• —Requires vLLM with compressed-tensors support
• —Not intended for vanilla Transformers inference of the quantized weights
• —Vision tower / adapter / projection remain BF16; only LM Linear weights are INT8
• —DFlash speculative drafter is not included or quantized in this release
• —Base model license: Apache 2.0