anemll/Qwen3.8-2.4T-A95B-FlashMoE-UD-Q1_0

Qwen3.8-2.4T-A95B Flash-MoE UD-Q1_0

SSD-backed Flash-MoE package for Qwen3.8-2.4T-A95B, exported from the Unsloth UD-Q1_0 GGUF.

This is not a conventional standalone GGUF. The non-routed and shared tensors are stored in model-dense.gguf; routed expert tensors are stored separately in a layer-major sidecar/ and loaded into a per-layer resident slot bank on demand. Use the Flash-MoE-enabled `Qwen3.8` branch of `anemll-flash-llama.cpp`. Stock llama.cpp builds that do not understand this package layout or its IQ1_XXXS routed experts cannot run it.

Model and package details

The routed sidecar preserves the source GGUF bytes exactly:

• —Layers 0-91: gate, up, and down routed tensors are IQ1_XXXS (276 tensors).
• —Layer 92: gate, up, and down routed tensors are Q2_K (3 tensors).
• —Each layer file contains the three whole routed tensors for that layer, with 512 experts packed within each tensor.

The dense/shared GGUF quant inventory is:

Download

The repository is about 397 GB. Put it on a fast local SSD and preserve the directory layout:

hf download anemll/Qwen3.8-2.4T-A95B-FlashMoE-UD-Q1_0 \
  --local-dir ~/Models/Qwen3.8

Expected layout:

~/Models/Qwen3.8/
├── model-dense.gguf
├── flashmoe-package.json
└── sidecar/
    ├── manifest.json
    ├── layer_000.bin
    ├── ...
    └── layer_092.bin

Keep the sidecar on fast directly attached storage. This runtime uses SSD reads for cache misses; network filesystems and slow external links will sharply reduce decode speed.

SSD slot-bank inference

Build the Flash-MoE branch of anemll-flash-llama.cpp, then run from its source directory:

./build/bin/llama-cli \
  -m ~/Models/Qwen3.8/model-dense.gguf \
  --moe-mode slot-bank \
  --moe-sidecar ~/Models/Qwen3.8/sidecar \
  --moe-slot-bank 96 \
  --moe-topk 10 \
  --moe-cache-io-split 4 \
  --moe-prefetch-temporal \
  -fit on \
  -ub 1 \
  -b 1 \
  -ngl 999 \
  -c 128 \
  --no-warmup \
  -st \
  -p "Hello" \
  -n 16

Important runtime choices:

• —Keep --moe-topk 10: this is the model's native routed width.
• —Do not add --slot4 or --slot8; those fused paths do not match native top-10 routing.
• —-ub 1 and -b 1 select the decode-oriented SSD path.
• —--moe-prefetch-temporal overlaps next-layer reads with current-layer compute.
• —The 96-slot setting is intended for a 128 GB Apple Silicon machine. Each slot, summed over all 93 layer banks, costs 703,856,640 bytes (671.25 MiB), so a 96-slot bank uses about 62.93 GiB in addition to the dense model and runtime allocations. If memory pressure is high, start with 32 or 64 slots.
• —The 262K value is model metadata, not a promise that the full context fits in memory. Increase -c only after budgeting KV-cache and runtime memory.

Attribution and license

• —Base model: Qwen/Qwen3.8-2.4T-A95B
• —Source quantization: unsloth/Qwen3.8-2.4T-A95B-GGUF, UD-Q1_0
• —Flash-MoE packaging/runtime: ANEMLL `anemll-flash-llama.cpp`, `Qwen3.8` branch

The embedded GGUF metadata reports license: other. Review and comply with the base model and source repository license terms before use or redistribution.

Verification metadata

flashmoe-package.json records the export summary and sidecar/manifest.json records source offsets, quant types, shapes, exact byte lengths, and repacked offsets. The package was produced with byte-level sidecar verification enabled.