hyperhuzaifa/Qwen3.6-35B-A3B-IK-K-GGUF

Qwen3.6-35B-A3B — ikllama IKK quants (GGUF)

Self-quantized IK_K GGUFs of Qwen/Qwen3.6-35B-A3B — a 35B Mixture-of-Experts (~3B active), so it decodes at small-model speed. Quantized with an importance matrix (qwen35moe.imatrix, included) using ik_llama.cpp's IK_K quant types, which give noticeably better quality-per-bit than standard K-quants on this MoE.

⚠️ These require [ik_llama.cpp](https://github.com/ikawrakow/ik_llama.cpp). The IQ*_K types are an ikllama extension and **will NOT load in mainline llama.cpp, LM Studio, Ollama, or koboldcpp.** If you need a portable quant, use a standard `Q4KM`/`Q5K_M` GGUF instead.

Files

IQ4K fits a single 24 GB GPU with room for context; IQ5K / IQ6_K are for more VRAM or dual-GPU.

Usage (ik_llama.cpp)

Build ik_llama.cpp, then:

./build/bin/llama-server \
  -m qwen35moe-IQ4_K.gguf \
  -ngl 99 -fa 1 -c 32768 \
  -ctk q8_0 -ctv q8_0 \
  -fmoe -rtr \
  --jinja --host 127.0.0.1 --port 8080

Flag notes (ik_llama dialect differs from mainline):

• —`-fa 1` — flash attention (not --flash-attn on).
• —`-fmoe` — ik_llama's fused-MoE kernel; meaningfully faster MoE decode. Recommended for this model.
• —`-rtr` — run-time tensor repack for your GPU; can speed up decode on some setups (optional).
• —Do NOT use `-mtp` — speculative decoding regresses on MoE (measured ~−28% on this model). These quants also ship no MTP head. Leave it off.
• —`-ub 512` (default) is fastest here; -ub 2048 is ~8% slower on this MoE.

Single-stream decode is very fast for a 35B thanks to the ~3B active params — on the order of 175–200+ t/s at IQ4_K on a single modern 24 GB GPU.

Credits & license

Base model: Qwen/Qwen3.6-35B-A3B, Apache-2.0. Quantized locally with ikllama.cpp + imatrix. Quant types courtesy of [ikllama.cpp](https://github.com/ikawrakow/ik_llama.cpp).