djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700

1---2license: mit3base_model: zai-org/GLM-5.3-Flash4pipeline_tag: image-text-to-text5library_name: transformers6tags:7  - glm8  - glm-59  - multimodal10  - vision-language11  - moe12  - sparse-attention13  - mla14  - quantization15  - rfa16  - rfi17  - rocm18  - rdna419  - gfx120120  - 4-bit21---22 23<div align="center">24 25# GLM-5.3-Flash · RFA + RFI8 composite quant (8× R9700 / RDNA4)26 27**A self-quantized derivative of [zai-org/GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash), tuned for 8× AMD Radeon R9700 (gfx1201 / RDNA4)**28 29[![Model size](https://img.shields.io/badge/params-321B%20(18B%20active)-blue)](https://huggingface.co/zai-org/GLM-5.3-Flash)30[![bpw](https://img.shields.io/badge/bpw-4.93-ff69b4)](#quantization)31[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](./LICENSE)32[![Serving stack](https://img.shields.io/badge/serving%20stack-tcclaviger%2Fvllm-00a2a2)](https://hub.docker.com/r/tcclaviger/vllm)33[![GPU](https://img.shields.io/badge/GPU-RDNA4%20(gfx1201)-ED1C24)](https://www.amd.com/en/products/processors/desktops/radeon.html)34 35</div>36 37---38 39## Model card40 41This is a **composite-quantized** checkpoint of Z.ai's **GLM-5.3-Flash** — the first natively42multimodal model in the GLM-5 series (321B total / ~18B active parameters). It was quantized from43the official FP8 release using the `tcclaviger/vllm` composite quantizer and is intended to be44served with **vLLM** on **RDNA4** (AMD Radeon R9700) hardware.45 46> **📦 Companion serving repo:** [GLM-5.3-Flash-rocm-r9700](https://huggingface.co/djdeniro/GLM-5.3-Flash-rocm-r9700) — the ROCm/RDNA4 overlay + quantization recipe.47 48### Benchmark49 50| Configuration | GPQA Diamond | Correct | Empty | Answered |51|---------------|---------|---|---|------|52| GLM-5.3-Flash (Z-AI API) | 80.8% | 76 | 6 | 94 | 53| GLM-5.3-Flash-RFA-RFI8 | 85.1% | 80 | 6 | 94 |54 55#### Details56 57 - Reasoning Effort: **high**58 - Max Output Len: 32k Tokens59 60 61***Total Context Limit for each task in test 32k, means 6x tasks use more than 32k output tokens***62 63### Serving performance (8× R9700, FY2026-09 production config)64 65| Scenario | Throughput |66|----------|------------|67| Decode, batch size 1, MTP OFF | ~34–39 tok/s |68| Decode, batch size 1, **MTP spec=3** | **~82–88 tok/s** |69| Aggregate, 4 concurrent, MTP spec=3 | ~155 tok/s |70| Context window (fp8 KV) | 300,000 tokens |71 72MTP speculative decoding: mean acceptance length ~3.7–3.9 of 4 draft tokens,73average draft acceptance 91–97% (live engine metrics, GPQA-style prompts).74 75 76---77 78## Table of contents79 80- [Attribution & credits](#attribution--credits)81- [Quantization](#quantization)82- [Quick start](#quick-start)83- [Model details](#model-details)84- [Multimodal policy](#multimodal-policy)85- [Known limitations](#known-limitations)86- [License](#license)87 88---89 90## Attribution & credits91 92| Component | Credit |93|-----------|--------|94| **Base model** | [zai-org/GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash) (Z.ai, MIT) |95| **Quantizer & quant kernels (RFA / RFI)** | [`tcclaviger/vllm:latest`](https://hub.docker.com/r/tcclaviger/vllm) (IronLLM Labs) |96| **RDNA4 port + quant recipe** | [GLM-5.3-Flash-rocm-r9700](https://huggingface.co/djdeniro/GLM-5.3-Flash-rocm-r9700) (this project's overlay) |97 98The composite **RFA + RFI8** quantization scheme, its kernels, and the serving runtime are provided99by **`tcclaviger/vllm` (IronLLM Labs)**. This checkpoint is the result of applying a quantization100*recipe* (archspec + source patches + kda-remap, in the companion repo) on top of that stack, and a101port of the GLM-5.3-Flash architecture to the RDNA4 serving path.102 103---104 105## Quantization106 107The checkpoint mixes three schemes in one `quant_method: "rfi"` composite:108 109| Scheme | Bits | Applied to | Stored size |110|--------|------|------------|-------------|111| **RFA** | 4.5 bpw | MoE routed experts (42 layers × 288 experts) | ~171.3 GB |112| **RFI8** | 8 bpw | attention / shared-expert / dense linears | ~7.8 GB |113| **BF16 / FP32** | 16 / 32 bpw | embeddings, vision tower, norms, MTP layer | ~18.7 GB |114 115### Average bits-per-weight116 117```118bpw = (total safetensors bytes × 8) / total parameters119    = (197,843,715,288 × 8) / 321,342,220,638120    = 4.9254  ≈  4.93 bpw121```122 123| Metric | Value |124|--------|-------|125| Total parameters | 321,342,220,638 (~321.3B) |126| On-disk size | 197.8 GB · 25 safetensors shards |127| **Average bpw** | **4.9254 ≈ 4.93** |128| vs. FP8 source | **0.60×** (197.8 GB vs 328.3 GB) |129| vs. BF16 | **0.31×** (197.8 GB vs 642.7 GB) |130 131---132 133## Quick start134 135```bash136docker pull tcclaviger/vllm:latest137 138git clone https://huggingface.co/djdeniro/GLM-5.3-Flash-rocm-r9700 overlay139 140# current production config (MTP spec=3, fp8 KV, 300k context)141docker run --rm --tty --ipc=host --shm-size=128g \142  --device /dev/kfd:/dev/kfd --device /dev/dri:/dev/dri \143  -v /path/to/GLM-5.3-Flash-RFA-RFI8-8xR9700:/models:ro \144  -v "$PWD/overlay":/overlay:ro \145  --entrypoint bash tcclaviger/vllm:latest \146  -c "/overlay/apply_overlay.sh && GLM5_NEXT_MTP_PROPOSER=1 exec vllm serve /models \147      --served-model-name glm53-flash --trust-remote-code --quantization rfi \148      --tensor-parallel-size 8 --gpu-memory-utilization 0.9575 \149      --max-model-len 300000 --max-num-seqs 4 --max-num-batched-tokens 2048 \150      --kv-cache-dtype fp8 \151      --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}' \152      --enable-prefix-caching --distributed-executor-backend mp \153      --compilation-config '{\"cudagraph_capture_sizes\":[1,2,4,8,16],\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"cudagraph_copy_inputs\":true}'"154```155 156---157 158## Model details159 160| Property | Value |161|----------|-------|162| Architecture | `Glm5NextForConditionalGeneration` |163| Layers | 45 = 34 KDA (linear attention) + 11 DSA (sparse-MLA) |164| Routed experts | 288 (top-8) + 1 shared expert |165| Extra | mHC hyper-connections, 1 nextn MTP draft layer, native vision tower |166| Context (fp8 KV) | 300,000 tokens |167 168---169 170## Multimodal policy171 172Images are resized with aspect ratio preserved, clamped to **min 384×384 / max 1280×1280**, and173fed with a min/max image-token budget. The model accepts image and video inputs natively.174 175---176 177## Known limitations178 179- **fp8 KV without runtime calibration** — serve with `--kv-cache-dtype fp8` and **scales fixed180  at 1.0**. Do not enable `--calculate-kv-scales`: runtime calibration on the profile dummy-run181  produces garbage scales from the uninitialized KDA recurrent state (details in the182  [serving repo](https://huggingface.co/djdeniro/GLM-5.3-Flash-rocm-r9700)).183- The 300k context / MTP spec=3 config presumes the VRAM headroom of the 256 GB 8× R9700 node.184- **Chat needs `reasoning_effort="low"`** — the default Reasoning Effort Max spends 16k+ tokens185  thinking before producing content on long generations.186 187---188 189## License190 191MIT. Base model © Z.ai (zai-org), MIT license. Quantizer & runtime © IronLLM Labs192([tcclaviger/vllm](https://hub.docker.com/r/tcclaviger/vllm)).193