neopolita/Qwen3.8-Flash-Next-113B-A5B-Niwaki-3bit-gguf

[image]

Qwen3.8-Flash-Next-113B-A5B-Niwaki-3bit-gguf

GGUF builds of [Qwen3.8-Flash-Next-113B-A5B-Niwaki-3bit-mlx](https://huggingface.co/neopolita/Qwen3.8-Flash-Next-113B-A5B-Niwaki-3bit-mlx) — Qwen3.8-Flash-Next pruned to 113B total / ~5.4B active parameters (routed experts kept on 32 of the 48 layers) — for llama.cpp on CUDA, Vulkan, Metal and CPU. Requires the [neopolita-llama.cpp](https://github.com/Neopolita/neopolita-llama.cpp) fork (see Usage): stock llama.cpp cannot represent this model's layout yet.

Niwaki (庭木): routed experts pruned using statistics of their routed tokens over a mixed web/code/chat/reasoning corpus, reconstructed to compensate, then briefly distilled from the full model, and stored at low precision. A paper with the full method is coming soon.

Files

Protocol: llama-perplexity, WikiText-2 test, 512-token windows. These llama.cpp numbers are not directly comparable to the MLX repo's 2048-window benchmarks; the canonical measurements (against the reference model) live there.

Both builds are quantized structure-aware, mirroring the MLX artifact's native allocation: the always-active backbone (attention, linear attention, hyper-connections, shared experts) at Q6K, embeddings and output at Q80, the pruned routed experts on a compact carrier (Q3K, or Q4K in the UD-Q4K file), the n-gram embedding table at Q2K (Q3K in UD-Q4K), and the distilled output maps at Q8_0.

Model dimensions

Usage

git clone https://github.com/Neopolita/neopolita-llama.cpp
cd neopolita-llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release   # add -DGGML_METAL=ON / -DGGML_CUDA=ON / -DGGML_VULKAN=ON as usual
cmake --build build -j --target llama-cli llama-server llama-completion

hf download neopolita/Qwen3.8-Flash-Next-113B-A5B-Niwaki-3bit-gguf Qwen3.8-Flash-Next-113B-A5B-Niwaki-3bit-UD-Q3K.gguf --local-dir .
./build/bin/llama-cli -m Qwen3.8-Flash-Next-113B-A5B-Niwaki-3bit-UD-Q3K.gguf
# or serve:
./build/bin/llama-server -m Qwen3.8-Flash-Next-113B-A5B-Niwaki-3bit-UD-Q3K.gguf

On an M4 Max (Metal) the UD-Q3K file decodes at ~35 tokens/s.

Why a fork

The fork adds three small, general features to llama.cpp's qwen4exp architecture, needed to store this model at its real size:

• —routed banks may be absent on a layer (the router and shared expert stay); stock llama.cpp needs uniform expert tensors, which would add ~17 GB of all-zero experts here;
• —a packed layout for the per-layer n-gram embedding table (ple.row_pack), so block-256 quantization types apply to its 160-wide rows; otherwise the table alone is 28.8 GB at 4.5 bits;
• —an optional per-layer output map (ffn_out_map), so the distilled healing maps ship as tensors instead of being folded into 3-bit weights.

Files produced by the fork's converter load only in the fork; upstreaming is planned.

Notes and limitations

• —Smaller sibling for 48 GB machines: Qwen3.8-Flash-Next-99B-A5B-Niwaki-3bit-gguf.

• —Quality follows the MLX artifact: ~99.8% of the reference's task average and 1.40× its perplexity under the MLX protocol. Fluent and on-topic; numbers and niche technical facts are less reliable than the reference.
• —Evaluated text-only on English-web-heavy data; the base model's biases are inherited.
• —Canonical benchmarks, dimensions and the MLX-native artifact: Qwen3.8-Flash-Next-113B-A5B-Niwaki-3bit-mlx.

Base model by the Qwen team (see license); pruning, distillation and GGUF builds by the Niwaki project, 2026-09.