CoolFace
Modelpublic

anemll/GLM-5.2-sidecar

sourceHugging Faceotherupdated 3mo agoView on Hugging Face
3likes162downloads
Model Card

GLM-5.2 Flash-MoE Sidecar (UD-IQ1_M)

SSD-streamed Mixture-of-Experts expert sidecar for GLM-5.2 (Unsloth Dynamic UD-IQ1_M), built for the Flash-MoE slot-bank runtime in the `anemll/flash-llama.cpp` fork.

The routed experts are stored as per-layer layer_*.bin files and streamed from SSD on demand into a small resident slot bank during decode, so the full MoE runs on a unified-memory Mac without keeping every expert in RAM. The dense / shared weights live in a separate small GGUF.

What's in this repo

PathSizeDescription
dense/model-dense.gguf~15.5 GBDense + shared weights, router, attention (the model you pass to -m)
dense/flashmoe-package.json—Flash-MoE package descriptor
layer_003.bin … layer_NNN.bin~213 GB totalPer-layer routed-expert tensors (IQ1_M), streamed on demand
manifest.json—Sidecar manifest (tensor map, quant types, byte offsets)

Model facts: arch glm-dsa, 256 routed experts, top-8 per token, 3 leading dense layers, n_embd = 6144, routed n_ff = 2048, experts quantized IQ1_M. Layout: layer_major_whole_tensor.

Total download is ~213 GB. You need a fast SSD; decode is I/O-bound on expert streaming.

Download

bash
hf download anemll/GLM-5.2-sidecar --repo-type model --local-dir ~/Models/GLM-5.2-sidecar

Build the runtime (Apple Metal)

This sidecar requires the Flash-MoE fork on the `GLM-5.2-Moe` branch:

bash
git clone -b GLM-5.2-Moe https://github.com/Anemll/anemll-flash-llama.cpp
cd anemll-flash-llama.cpp
cmake -B build -DGGML_METAL=ON
cmake --build build --config Release -j --target llama-cli

Run / test

bash
./build/bin/llama-cli --perf \
  -m ~/Models/GLM-5.2-sidecar/dense/model-dense.gguf \
  --moe-mode slot-bank \
  --moe-sidecar ~/Models/GLM-5.2-sidecar/ \
  --moe-verify-sidecar \
  --moe-slot-bank 64 \
  --moe-topk 8 \
  --moe-cache-io-split 2 \
  --moe-prefetch-temporal \
  -fit on \
  -ub 1 -b 64 \
  -ngl 999 \
  -c 512 \
  --seed 123 --temp 0 \
  -p "What is Apple Neural Engine? Answer in one sentence." \
  -n 2000 -st \
  --slot8

--slot8 (fused single-kernel routed FFN)

This branch adds --slot8, which collapses the whole routed FFN — gate, up, SwiGLU, down, and the routed weighted-sum over all selected experts — into a single fused op (two Metal kernels, IQ1M) for single-token decode. It reads the resident slot ids once at encode time, so the per-expert `mulmat_id` decode replay / ICB cache is no longer used on that path. Output is validated byte-identical to the unfused reference path.

Toggles:

  • —--slot8 / --no-slot8 — enable/disable the fused path (only engages on eligible top-k decode layers).
  • —LLAMA_FLASH_MOE_SLOT8_REFERENCE=1 — force the mul_mat reference path (A/B comparison / fallback).
  • —LLAMA_FLASH_MOE_SLOT8_DEBUG=1 — log which layers take the fused path.
Tested on Apple M5 Max (128 GB). --slot8 is a decode-only fast path; prefill and non-eligible layers use the normal slot-bank route.

License

Derived from GLM-5.2 (Z.ai / Zhipu AI). Use is subject to the original GLM-5.2 model license; this sidecar only repackages those weights for SSD-streamed inference.