neopolita/Qwen3.6-23B-A3B-Niwaki-v2.3-4bit-gguf

[image]

Qwen3.6-23B-A3B-Niwaki-v2.3-4bit-gguf

GGUF builds of [Qwen3.6-23B-A3B-Niwaki-v2.3-4bit-mlx](https://huggingface.co/neopolita/Qwen3.6-23B-A3B-Niwaki-v2.3-4bit-mlx) — Qwen3.6-35B-A3B pruned to 22.6B total / ~3.8B active parameters, with the model's program-state tracking and few-shot reasoning intact — for llama.cpp on CUDA, Vulkan, Metal and CPU. Requires the [neopolita-llama.cpp](https://github.com/Neopolita/neopolita-llama.cpp) fork (see Usage): stock llama.cpp cannot represent this model's layout yet.

Niwaki (庭木) are Japan's garden trees, sculpted by meticulous pruning so that every branch serves the form of the whole. This model applies that spirit to a Mixture-of-Experts. A paper with the full method and measurements is coming soon.

Project page, with the releases and their measurements side by side: [niwakiai.com](https://niwakiai.com/)

Files

v2.3 keeps v2.2's layout and sizes exactly; what changed is only the recovery training that follows the pruning. The earlier generations' GGUF builds under the identical protocol:

This build beats the v2.2-4bit UD-Q3K build (8.61) by 2.5% at the same size (5% on the C4 slice), the v2.1 build by 8%, and the first-generation flagship by 19% at 2.3 GB fewer bytes.

Reference Qwen3.6-35B-A3B at Q8_0 measures 6.95 under the identical protocol (llama-perplexity, WikiText-2 test, 512-token windows). These llama.cpp numbers are not directly comparable to the MLX repo's 2048-window benchmarks; the relative standings match across both, and the canonical measurements (task suites, gsm8k, the program-state battery) live there.

Generation battery (measured on the canonical MLX weights; reference scores 0.63 / 0.51 under the identical battery): bigram-diversity avg/min = 0.64 / 0.48 across an 8-prompt code/reasoning/chat/creative battery.

The recommended UD-Q3K build is quantized structure-aware (importance matrices calibrated on a mixed web/code/chat/reasoning corpus), mirroring the artifact's native allocation: the always-active backbone (attention, shared experts, embeddings) is kept at high precision (Q6K), the untouched layers' routed experts ride a compact carrier (q3k, imatrix-guided), and the 64-expert banks of layers 10–29 — the part the MLX artifact never takes below 4-bit either — stay at q4k. It stays within 1% of uniform Q4KM quality at ~20% fewer bytes; pick Q4K_M if you want the last percent of quality over size.

Format note: no zero-valued pad: layers 10–29 store exactly their 64 kept experts (the fork reads a per-layer expert count), so the file carries only experts that exist.

Speed (M4 Max, Metal, llama-bench, 512-token prompt / 128 generated; measured on the v2.2 builds, whose file layout and sizes these files share exactly): UD-Q3K 1,430 tok/s prompt, 82 tok/s decode; Q4KM 1,475 / 91 tok/s — the Q4K kernels are faster than Q3K on Metal, so Q4KM is the quicker file as well as the slightly better one.

Against the intact model at the same size

The alternative to pruning is to quantize the intact 35B model harder. Unsloth's dynamic quants of Qwen3.6-35B-A3B that bracket these files, measured on the same machine, build and protocol (llama-perplexity, 512-token windows, WikiText-2 test and a 512-chunk C4 slice; llama-bench 512-token prompt / 128 generated, Metal):

At equal size the intact model's dynamic quants read 8–11% lower WikiText-2 and 10–13% lower C4 perplexity than these builds; these builds run 1.7–2.1× faster on Metal, on prompt processing and decode alike (the Q3K / Q4K kernels against the IQ kernels, and fewer routed-expert bytes per token). Pick by what you are short of: quality at equal memory favours the intact quants, throughput at equal memory favours these files. The function-level comparison (task suites, gsm8k, program-state battery) is on the MLX card.

Model dimensions

Usage

git clone https://github.com/Neopolita/neopolita-llama.cpp
cd neopolita-llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release   # add -DGGML_METAL=ON / -DGGML_CUDA=ON / -DGGML_VULKAN=ON as usual
cmake --build build -j --target llama-cli llama-server llama-completion

hf download neopolita/Qwen3.6-23B-A3B-Niwaki-v2.3-4bit-gguf Qwen3.6-23B-A3B-Niwaki-v2.3-4bit-UD-Q3K.gguf --local-dir .
./build/bin/llama-cli -m Qwen3.6-23B-A3B-Niwaki-v2.3-4bit-UD-Q3K.gguf
# or serve:
./build/bin/llama-server -m Qwen3.6-23B-A3B-Niwaki-v2.3-4bit-UD-Q3K.gguf

Why a fork

Layers 10–29 of this model keep 64 of their 256 routed experts and route only over them. Stock llama.cpp requires every MoE layer of a model to carry the same number of experts; the fork lets a qwen35moe layer's expert count follow its tensors, with a router of one row per kept expert (its softmax is exactly the renormalised routing of the MLX artifact). Files produced by the fork's converter load only in the fork; upstreaming is planned.

Canonical benchmarks, the progress table since the first generation, and the MLX-native artifact: Qwen3.6-23B-A3B-Niwaki-v2.3-4bit-mlx. Family: v2.3-2bit · v2.2 4-bit / 2-bit · v2.1 4-bit / 2-bit · v2 4-bit / 2-bit · first-generation 27B-2bit · 19B-2bit · 11B-4bit.

Base model by the Qwen team (Apache 2.0); pruning, recovery training, and GGUF builds by the Niwaki project, 2026-09.