r0b0tlab/GLM-5.3-Flash-EXL3-2.25bpw-sm121

GLM-5.3-Flash EXL3 2.25bpw (GB10 / SM121 runtime)

EXL3 (ExLlamaV3 trellis) quantization of `zai-org/GLM-5.3-Flash` (320B total / ~18B active MoE) by r0b0tlab, produced with the converter contract of the `r0b0tlab/glm53-flash-exl3-dflash2-sm121` runtime (vLLM v0.30.0rc1 + EXL3 plugin + anchored patches + vendored EXL3 kernels).

• —Format: EXL3 (trellis storage, exllamav3-compatible), ~2.25 bpw average, mul1 codebook, mixed K3/K4 tiles, output scales on, 6-bit head.
• —Size: 98.5 GB (91.8 GiB) across 31 safetensors shards, plus quantization_config.json (full module/tensor manifest) and the tokenizer/template files. 42 files total.
• —Calibration: mixed corpus, tap-K3 selection; details in quantization_config.json.

Runtime — there is no container image

No container image is published for this pack, and the measurements below were not produced inside one. There is no ghcr.io/r0b0tlab/vllm-exl3-sm121 and no GHCR package for the runtime repo. The docker/Dockerfile.stack in that repo is an unbuilt template whose base is still a placeholder, and no published vLLM v0.30.0rc1 container image exists to point it at.

The measurements come from a host-native install on a single GB10: a vLLM source checkout at a00a3544b93e (v0.30.0rc1), the EXL3 plugin installed editable, and its CUDA extension built in place — served under systemd-run --user --scope. To reproduce, use the host bootstrap in the runtime repo (scripts/bootstrap_cluster.sh), not a docker pull.

Serving (single NVIDIA GB10 / DGX Spark)

Runtime source: `r0b0tlab/glm53-flash-exl3-dflash2-sm121` — runtime.lock.json, patches/vllm/, docs/RESULTS.md.

vllm serve r0b0tlab/GLM-5.3-Flash-EXL3-2.25bpw-sm121 \
  --quantization exl3 --trust-remote-code \
  --served-model-name glm53-flash-exl3-dflash2 \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.85 --max-model-len 32768 --max-num-seqs 16 \
  --reasoning-parser glm47 --no-async-scheduling --max-num-batched-tokens 2048

Add DFlash2 speculative decoding (draft: incoai/GLM-5.3-Flash-DFlash2, converted to EXL3 locally — the draft is CC-BY-NC-ND and is not redistributed here, so you must supply your own path):

  --speculative-config '{"method":"dflash","model":"<local-dflash2-exl3>","num_speculative_tokens":5}' \
  --compilation-config '{"cudagraph_capture_sizes":[1,2,4,8,16,24,32,48,56,64,80,96,112,128]}' \
  --enable-auto-tool-choice --tool-call-parser glm47

num_speculative_tokens: 5 and that cudagraph_capture_sizes list are the measured K=5 configuration.

Measured (single GB10, host-native, TP=1, temp 0, DFlash2 K=5, 2026-09-20)

(K=7 was also measured: ~4 % faster on structured single-stream but slower at concurrency — 49.8 agg at ×16 vs 67.1. K=5 is the shipped choice.)

Losslessness, acceptance telemetry and the engine receipts: `r0b0tlab/glm53-flash-exl3-dflash2-sm121` — docs/RESULTS.md, docs/M4-RECEIPT.md, docs/CONCURRENCY-FINDING.md.

Known limits

• —Single GB10, TP=1. C≥2 tool-call loads not exercised.
• —Vision: verified by an image Q&A smoke on this config, but image requests are drafted text-only (the DFlash2 drafter does not accept external multimodal embeddings).
• —The concurrency ceiling is the vLLM v1 host step on this platform (~2 ms per D2H completion), not the kernels.

Provenance

Quantized by r0b0tlab from zai-org/GLM-5.3-Flash (MIT). Weights are unchanged beyond quantization. EXL3 format follows exllamav3 v1.5.0 (MIT); the runtime is Apache-2.0 (vLLM) + MIT (exllamav3).