Mitchins/GLM-4.6V-Flash-NVFP4-BF16Vision

GLM-4.6V-Flash · NVFP4 BF16-Vision (compressed-tensors)

Experimental. NVFP4 is a Blackwell (RTX 50xx / GB2xx) native format. Inference on older hardware may fall back to emulated paths or fail.

Quantized release of `zai-org/GLM-4.6V-Flash` using compressed-tensors NVFP4, with the entire multimodal vision stack (model.visual.*) preserved in BF16 and verified with end-to-end image understanding tests.

See also: `Mitchins/GLM-4.6V-Flash-FP8-Block128` for the more broadly compatible FP8 variant.

What was quantized

Calibration: 256 samples from wikitext-2-raw-v1 (train split), max 512 tokens, concatenated. Text-only calibration is sufficient because the vision stack is excluded from quantization.

240 language-model Linear layers quantized; 181 vision tensors kept at BF16.

Disk size

Verification

Tested on an NVIDIA RTX 5090 (32 GB) with:

• —transformers==4.57.6
• —llmcompressor==0.10.0.1
• —vllm==0.19.0
• —torch==2.10.0+cu128
• —CUDA 12.8

Bridge image test (Sydney Harbour Bridge, Wikimedia Commons):

"The image shows the Sydney Harbour Bridge at night. The bridge is illuminated with lights along the bridge. The background shows the city skyline with buildings illuminated at night. The Sydney Harbour Bridge is a large steel arch bridge that spans Sydney Harbour."

Model correctly identified the bridge, arch structure, and night-time city context. Vision stack functional end-to-end.

Usage

vLLM (recommended for RTX 5090)

vllm serve Mitchins/GLM-4.6V-Flash-NVFP4-BF16Vision --dtype bfloat16

The server auto-detects quantization: compressed-tensors from the config.

Transformers (with compressed-tensors)

import torch
from PIL import Image
from transformers import (
    Glm4vForConditionalGeneration,
    Glm4vImageProcessor, Glm4vVideoProcessor, Glm4vProcessor,
    AutoTokenizer,
)

MODEL = "Mitchins/GLM-4.6V-Flash-NVFP4-BF16Vision"

model = Glm4vForConditionalGeneration.from_pretrained(
    MODEL, dtype=torch.bfloat16, device_map="auto"
)

# Patch rope_scaling for transformers: config stores [8,12,12] (vllm-compatible),
# but transformers doubles it internally so we must pre-double to [16,24,24].
_rs = model.config.text_config.rope_scaling
if _rs and _rs.get("mrope_section") == [8, 12, 12]:
    _tf = [x * 2 for x in _rs["mrope_section"]]
    for _mod in model.modules():
        if hasattr(_mod, "rope_scaling") and isinstance(getattr(_mod, "rope_scaling", None), dict):
            _mod.rope_scaling = {**_mod.rope_scaling, "mrope_section": _tf}

tokenizer       = AutoTokenizer.from_pretrained(MODEL)
image_processor = Glm4vImageProcessor.from_pretrained(MODEL)
video_processor = Glm4vVideoProcessor.from_pretrained(MODEL)
processor = Glm4vProcessor(
    image_processor=image_processor,
    tokenizer=tokenizer,
    video_processor=video_processor,
    chat_template=tokenizer.chat_template,
)

image = Image.open("your_image.jpg").convert("RGB")
messages = [{"role": "user", "content": [
    {"type": "image"},
    {"type": "text", "text": "Describe this image."},
]}]
text   = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=text, images=[image], return_tensors="pt")
inputs = {k: v for k, v in inputs.items() if k != "token_type_ids"}
inputs = {k: v.to(model.device) for k, v in inputs.items()}

with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=256, do_sample=False)

print(processor.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Known caveats and packaging fixes

All caveats from the FP8 repo apply here. NVFP4-specific additional notes:

5. NVFP4 requires calibration data

Unlike FP8_BLOCK (data-free), NVFP4 uses SequentialPipeline in llmcompressor and requires a calibration dataloader. Pass a dataset to oneshot():

oneshot(
    model=model, recipe=recipe, output_dir=SAVE_DIR,
    dataset="wikitext",
    dataset_config_name="wikitext-2-raw-v1",
    splits="train",
    num_calibration_samples=256,
    max_seq_length=512,
    text_column="text",
    concatenate_data=True,
)

Text-only calibration is sufficient because model.visual.* is excluded.

6. NVFP4 packs weights in-place — cannot run transformers forward after oneshot

After oneshot() with NVFP4, the in-memory model has its Linear.weight replaced by packed representations. Standard model.generate() raises AttributeError: 'Linear' object has no attribute 'weight'. Reload from the saved directory before running inference:

del model; torch.cuda.empty_cache()
model = Glm4vForConditionalGeneration.from_pretrained(SAVE_DIR, dtype=torch.bfloat16, device_map="auto")

Vision tower, ropescaling, AutoProcessor, tokentype_ids

See fixes 1–4 in the FP8 repo README. All apply identically here, including the in-memory mrope_section doubling required for transformers inference (fix 2).

Reproducibility

python compress.py --fp4

Full source: `compress.py` and `verify.py` are included in this repo.

License

Derived from `zai-org/GLM-4.6V-Flash`, released under MIT. This quantized derivative inherits the same license.