RESMP-DEV/GLM-4.7-Flash-Marlin-MMFP4

GLM-4.7-Flash-Marlin-MMFP4

MMFP4-quantized GLM-4.7-Flash — a 30B-A3B MoE model compressed to 4 bits per weight using GPTQ with actorder and Metal Marlin's E2M1 FP4 format.

Model Description

This is a quantized version of zai-org/GLM-4.7-Flash, the strongest model in the 30B class that balances performance and efficiency.

GLM-4.7-Flash features:

• —30B-A3B MoE architecture (64 experts + shared expert, 2-4 active per token)
• —Multi-head Latent Attention (MLA) for 8× KV cache compression
• —State-of-the-art reasoning (91.6% on AIME 2025, 59.2% on SWE-bench Verified)
• —Bilingual (English + Chinese)

Quantization Details

Quantized using MR-GPTQ (Metal Marlin GPTQ) with CUDA acceleration:

Method

• —Format: MMFP4 (E2M1 FP4) — Metal Marlin's native FP4 format
• —Quantization: GPTQ with actorder (activation-order column permutation)
• —Hessian calibration: Pre-computed Hessians for attention layers
• —Expert quantization: Identity Hessian with actorder (no calibration data for MoE experts)
• —Group size: 128
• —Hardware: NVIDIA RTX 3090 Ti (CUDA-accelerated Cholesky factorization)

Quantization Statistics

• —Total tensors: 19,066
• —Shards: 48 safetensors files
• —Quantization time: ~20 minutes (RTX 3090 Ti)

Files

GLM-4.7-Flash-Marlin-MMFP4/
├── model-00001-of-00048.safetensors   # Layer 0 (embeddings)
├── model-00002-of-00048.safetensors   # Layer 1
├── ...
├── model-00048-of-00048.safetensors   # Layer 47 + lm_head
├── model.safetensors.index.json       # Weight map
├── config.json                        # Model config
├── generation_config.json
├── tokenizer.json                     # Tokenizer
└── tokenizer_config.json

Usage

With Metal Marlin (Apple Silicon)

from metal_marlin import MarlinForCausalLM
from transformers import AutoTokenizer

model = MarlinForCausalLM.from_pretrained(
    "RESMP-DEV/GLM-4.7-Flash-Marlin-MMFP4",
    device="mps"
)
tokenizer = AutoTokenizer.from_pretrained("zai-org/GLM-4.7-Flash")

prompt = "<|user|>\nExplain quantum computing in simple terms.\n<|assistant|>\n"
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to("mps")
output = model.generate(input_ids, max_new_tokens=256, temperature=0.7)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Tensor Format

Each quantized weight tensor has corresponding scale factors:

• —{name}.weight: Packed FP4 weights (uint8)
• —{name}.scales: FP16 per-group scales (group_size=128)

Hardware Requirements

Benchmarks

Original Model Performance (from Z.AI)

Quantized Model Notes

• —GPTQ with actorder minimizes quality loss vs RTN
• —Expected degradation: ~1-2% on benchmarks vs FP16
• —E2M1 FP4 format optimized for Metal Performance Shaders

Comparison with Trellis Quant

Choose Trellis for smaller size, MMFP4 for simpler tensor format and potentially better compatibility.

Limitations

• —Metal Marlin required for optimal inference on Apple Silicon
• —No speculative decoding yet
• —Quality loss: ~1-2% on benchmarks vs FP16 (typical for 4-bit quantization)

Credits

• —Original model: Z.AI / GLM Team
• —Quantization method: GPTQ with actorder
• —Quantization toolkit: Metal Marlin

Citation

If you use this model, please cite the original GLM-4.5 paper:

@misc{glm2025glm45,
      title={GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models}, 
      author={GLM Team and Aohan Zeng and Xin Lv and others},
      year={2025},
      eprint={2508.06471},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2508.06471}, 
}

License

This quantized model inherits the MIT License from the original GLM-4.7-Flash model.