Modelpublic

aj9o9/Qwen3.6-35B-A3B-Escha-W2-GGUF

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face

9likes230downloads

Model Card

Escha Qwen3.6-35B-A3B W2 GGUF

I ported EschaLabs/Qwen3.6-35B-A3B-Escha-W2 to llama.cpp. Weights are theirs. This is not a requant.

I decode their native 2-bit eschamoe code in-kernel (GGML_OP_ESCHA_MOE). Expert payloads are the same bytes as the safetensors. Non-expert tensors are fp16.

Stock llama.cpp cannot load this file. You need my fork, branch escha-w2:

https://github.com/Ajay9o9/llama.cpp-escha/tree/escha-w2

If you only download the GGUF, it will not run.

File

	Source checkpoint	This GGUF
Size	12.30 GB	14.50 GB (+17.9%)
Expert code	native 2-bit / 3-bit	same bytes
Non-expert tensors	int8	fp16
Layers	40 + MTP	40, MTP dropped
Tensors		976
VRAM, 32k context		14.5 GB of 24

File: Escha-Qwen3.6-35B-W2.gguf. 256 experts, top-8. gate/up K=2, down K=3. Extra size is fp16 on the non-expert tensors, not the expert code.

Quality vs Escha SGLang

Same GPU. I sent token ids, not text. Tokenizers agreed on 196,608 tokens of wikitext. Perplexity uses the same chunking on both sides (n_ctx 2048, 96 chunks, 98,208 scored tokens).

Perplexity, wikitext-2
Escha SGLang	6.0988
This GGUF	6.0997 +/- 0.047
Difference	+0.0009 (+0.015%)

Gap is inside the llama.cpp run's own error bar, and smaller than SGLang's own rerun drift (+0.0016). One corpus. Not a task-accuracy claim.

Teacher-forced, 97 positions
Top-1 token agreement	93 / 97 = 95.9%
JS divergence over top-20	mean 1.7e-4, max 1.0e-3
\	delta logprob\	where they agree	mean 0.013, max 0.072
True-token NLL	SGLang 1.39704, this GGUF 1.38663

Same prefix at every position, so a miss at i-1 does not poison i. Two SGLang runs against each other were 96 / 97.

Greedy, 16 prompts, 64 tokens, temp 0
Token-identical	13 / 16
Diverged	3 / 16, at tokens 5, 26, 49

Two SGLang runs were also 13 / 16 identical. Greedy is a hard argmax. I would not hang a quality claim on this.

Op test vs a numpy dense-fold reference (CPU and CUDA scored separately): rel RMS 1.06e-6 at K=2, 9.26e-7 at K=3.

Speed vs Escha SGLang

Single stream, batch 1, one RTX 3090. Their runtime is still ~2.6x faster at generation. Context lengths were not matched, so the ratio is rough.

	Escha SGLang	This GGUF
Generation (tok/s)	110.4	42.6
Prefill, 4,566 tokens (tok/s)	2595	634

SGLang generation was 256 new tokens with ignore_eos. MTP is dropped here, so speculative decoding on their server is not in these numbers.

Build the fork

git clone -b escha-w2 https://github.com/Ajay9o9/llama.cpp-escha.git
cd llama.cpp-escha
cmake -S . -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86
cmake --build build -j 12

86 is an RTX 3090. Change that for your GPU. -j 12 is a 12-core CPU.

Download the GGUF

hf download aj9o9/Qwen3.6-35B-A3B-Escha-W2-GGUF Escha-Qwen3.6-35B-W2.gguf

Direct file: https://huggingface.co/aj9o9/Qwen3.6-35B-A3B-Escha-W2-GGUF/resolve/main/Escha-Qwen3.6-35B-W2.gguf

Run it

./build/bin/llama-server \
  -m Escha-Qwen3.6-35B-W2.gguf \
  -ngl 99 --host 127.0.0.1 --port 8080 \
  -c 32768 -np 1 -t 12 --jinja

Or:

./build/bin/llama-cli \
  -m Escha-Qwen3.6-35B-W2.gguf \
  -ngl 99 --jinja -c 32768

It is a reasoning model. Give it room to think or you get an empty answer.

What this is not

—Not a Q4K / Q80 requant of a dense reconstruction
—Not upstream llama.cpp
—Not bit-identical to Escha SGLang on every greedy prompt
—No MMLU / GSM8K / long-context eval

License

Apache-2.0, same as the Escha weights and the escha 1.0.2+qwen3moe wheel I used.

Source: EschaLabs/Qwen3.6-35B-A3B-Escha-W2 Runtime I compared against: escha 1.0.2+qwen3moe on SGLang