Modelpublic

RDson/Qwen3.6-35B-A3B-IQ4_KS-GGUF

sourceHugging Faceapache-2.0updated 5mo agoView on Hugging Face

4likes154downloads

Model Card

Qwen3.6-35B-A3B IQ4_KS GGUF

ik_llama.cpp is required to run this model.

ik_llama.cpp imatrix quantization of Qwen/Qwen3.6-35B-A3B.

This quant uses a mixed-precision recipe to maximize quality while fitting entirely into 24GB VRAM for full GPU offloading. Attention, SSM, and shared expert layers are kept at Q8_0/F32, while the massive routed MoE expert layers are quantized to IQ5_KS and IQ4_KS.

Perplexity

Measured against wiki.test.raw with n_ctx=512:

Quant	Size	PPL
IQ4_KS	~19.8 GiB	6.7401 +/- 0.04381

*Note: Lower is better.

Quantization Recipe

The "Secret Recipe" used for the mixed-precision tensor overrides is based on the methodology used by ubergarm:

bash

custom="
# 60 Repeating Layers [0-59]
## Gated Attention/Delta Net [Blended 0-59]
blk\..*\.attn_gate\.weight=q8_0
blk\..*\.attn_qkv\.weight=q8_0
blk\..*\.attn_output\.weight=q8_0
blk\..*\.attn_q\.weight=q8_0
blk\..*\.attn_k\.weight=q8_0
blk\..*\.attn_v\.weight=q8_0
blk\..*\.ssm_alpha\.weight=f32
blk\..*\.ssm_beta\.weight=f32
blk\..*\.ssm_out\.weight=q8_0
# Shared Expert Layers [0-59]
blk\..*\.ffn_down_shexp\.weight=q8_0
blk\..*\.ffn_(gate|up)_shexp\.weight=q8_0
# Routed Experts Layers [0-59]
blk\..*\.ffn_down_exps\.weight=iq5_ks
blk\..*\.ffn_(gate|up)_exps\.weight=iq4_ks
# Non-Repeating Layers
token_embd\.weight=q8_0
output\.weight=q8_0
"

custom=$(
  echo "$custom" | grep -v '^#' | \
  sed -Ez 's:\n+:,:g;s:,$::;s:^,::'
)

How This Was Made (Reproduction Steps)

Because generating an imatrix directly from a ~65GB BF16 model requires massive system RAM, we use a Q8_0 intermediate step to generate the imatrix without running out of memory.

1. Convert HF Safetensors to BF16 GGUF

bash

python llama.cpp/convert_hf_to_gguf.py \
  --outtype bf16 \
  --split-max-size 50G \
  --outfile ./Qwen3.6-35B-A3B-BF16.gguf \
  /path/to/Qwen3.6-35B-A3B/

2. Quantize to Q8_0 (For Imatrix Generation)

bash

./ik_llama.cpp/build/bin/llama-quantize \
  ./Qwen3.6-35B-A3B-BF16-00001-of-00002.gguf \
  ./Qwen3.6-35B-A3B-Q8_0.gguf \
  Q8_0 16

3. Generate the Imatrix Note: `GGML_CUDA_NO_PINNED=1` is used to prevent system RAM exhaustion on 24GB VRAM setups.

bash

GGML_CUDA_NO_PINNED=1 ./ik_llama.cpp/build/bin/llama-imatrix \
  -m ./Qwen3.6-35B-A3B-Q8_0.gguf \
  -f /path/to/ubergarm-imatrix-calibration-corpus-v02.txt \
  -o Qwen3.6-35B-A3B-imatrix.dat \
  --ctx-size 512 \
  -t 11 \
  --fit

4. Quantize BF16 to IQ4_KS We use the original BF16 model here with the Q8_0-generated imatrix for maximum fidelity.

bash

./ik_llama.cpp/build/bin/llama-quantize \
  --imatrix ./Qwen3.6-35B-A3B-imatrix.dat \
  --custom-q "$custom" \
  ./Qwen3.6-35B-A3B-BF16-00001-of-00002.gguf \
  ./Qwen3.6-35B-A3B-IQ4_KS.gguf \
  IQ4_KS 16

5. Test Perplexity

bash

wget https://huggingface.co/datasets/ikawrakow/validation-datasets-for-llama.cpp/resolve/main/wiki.test.raw.gz
gunzip wiki.test.raw.gz

./ik_llama.cpp/build/bin/llama-perplexity \
  -m ./Qwen3.6-35B-A3B-IQ4_KS.gguf \
  -f ./wiki.test.raw \
  -c 512 \
  -ngl 99 \
  -t 1 \
  -fa

Quick Start Inference

Requires ik_llama.cpp.

bash

./ik_llama.cpp/build/bin/llama-server \
  -m ./Qwen3.6-35B-A3B-IQ4_KS.gguf \
  -c 131072 \
  -ngl 99