amd/Qwen3.6-35B-A3B-Quark-UINT4-G32-GGUF

Qwen3.6-35B-A3B · uint4wo32 → GGUF Q4_1

Base model

Quantization

Reproduce quantization

This directory provides a one-shot entry point for Q41: after Quark `uint4wo32` quantization, the GGUF Q41 is exported directly from memory (by default no intermediate safetensors is written to disk).

cd /home/l/work/quantization_work/quark_llama_cpp_test/models/qwen3.6-35b-a3b
./scripts/quantize_q4_1.py

By default the GGUF is written to gguf-q4_1/. You can override the default paths and device via BASE_MODEL, GGUF_OUT, QUARK_SRC, LLAMA_CPP_DIR, DEVICE, MULTI_GPU=1. To quantize only and skip GGUF export, add --skip-gguf-export.

GGUF export

Export paths (hybrid, experimental)

Known limitation: linear_attn / shared_expert are re-quantized during export; a future improvement is to make the whole path native passthrough.

Files

gguf-q4_1/
├── qwen36-35b-a3b-uint4-q4_1-q4_1-q4_1-00001-of-00003.gguf  (7.4 GB)
├── qwen36-35b-a3b-uint4-q4_1-q4_1-q4_1-00002-of-00003.gguf  (7.4 GB)
└── qwen36-35b-a3b-uint4-q4_1-q4_1-q4_1-00003-of-00003.gguf  (6.9 GB)

Evaluation: GSM8K

Results

Comparison: old unaligned protocol

The old script used a plain Answer: completion with max_tokens=512 and scored on the last number in the generated text. Under that protocol the result was 70.66% (931 / 1319), mainly affected by Qwen3.6's thinking output and the scoring method; it is not a fair direct comparison against the gsm8k_nothink result.

Reproduce

cd /home/l/work/quantization_work/quark_llama_cpp_test/models/qwen3.6-35b-a3b
./scripts/run_gsm8k_nothink.sh

# smoke test example
NUM_QUESTIONS=10 ./scripts/run_gsm8k_nothink.sh

lm-eval compatible config: scripts/gsm8k_nothink.yaml. Logs: logs/gsm8k_eval.log, logs/llama_server_gsm8k.log.

Inference

PROFILE=qwen3.6-35b ./run_llama.sh

# or specify a shard
GGUF=gguf-q4_1/qwen36-35b-a3b-uint4-q4_1-q4_1-q4_1-00001-of-00003.gguf ./run_llama.sh

2026-07-21 single-machine smoke test: llama-cli loaded successfully, generating ~45 tok/s.

Notes

• —The uint4-wo32/ directory here is currently a symlink to an earlier quantization artifact; the GGUF and evaluation results in this directory are authoritative.
• —The GSM8K score is a Q4_1 hybrid export test baseline, not an official BF16 reference; the scoring protocol may differ slightly from lm-eval flexible-extract.
• —Qwen3.6 emits thinking output by default, which affects token usage and last-number extraction accuracy.

Evaluation: Perplexity (wikitext-2, llama.cpp)

All three were run with llama.cpp llama-perplexity on wikitext-2 test (n_ctx=512, 20 chunks, same data and same parameters), so they are directly comparable.

Quantization is nearly lossless: Q41 is only +1.8% over F16 and Q40 only +2.7%, while the size is compressed to about 30% of the float model. Before fixing the linear_attn V-head reordering, Q4_0 PPL was 2,619,116; after the fix it is 6.79.

Evaluation: Open LLM Leaderboard v2

How all 6 tasks are measured on the GGUF

Update (2026-08-06): An earlier version of this card stated that only the 2 generative tasks (IFEval, MATH-hard) could be scored on the GGUF. That limitation has since been lifted by patching llama-server; the four multiple-choice tasks are now measured directly on this GGUF file (see below).

The six leaderboard-v2 tasks use two scoring methods in lm-eval:

• —Generative (`generate_until`): IFEval, MATH-hard. The model generates text freely, which is then matched against rules or the reference answer. Any backend that can generate can score these.
• —Multiple-choice (`multiple_choice` / loglikelihood): BBH, GPQA, MMLU-Pro, MuSR. Each candidate answer is appended to the question, and the model computes the sum of the per-position log-probabilities of that already-fixed token sequence, then picks the highest-scoring candidate. This requires the backend to return the logprob of the input (prompt) tokens.

Stock llama-server returns logprobs only for tokens the model itself generates, not for prompt tokens (both /v1/completions with echo=true and the native /completion with n_probs were tested), so the four multiple-choice tasks originally could not be scored. This was fixed by patching llama-server on branch `hongweimeng/gguf-prompt-logprobs`: /v1/completions with echo=true + logprobs now returns the legacy OpenAI logprobs format (text_offset / tokens / token_logprobs / top_logprobs) covering the prompt tokens. With that, lm-eval's gguf (GGUFLM) backend scores all four multiple-choice tasks directly on the GGUF. Logprob correctness was verified against llama-perplexity (matching PPL).

So all 6 leaderboard-v2 tasks are now measured on this GGUF file — no vLLM / safetensors substitution is needed.

Results (updated 2026-08-08, all 6 tasks complete)

Backend for this GGUF: the 2 generative tasks (IFEval, MATH-hard) use lm_eval --model local-completions -> llama-server (/v1/completions, -c 32768 -np 4, num_concurrent=4), measured 2026-08-04. The 4 multiple-choice tasks use lm-eval's gguf (GGUFLM) backend against the patched llama-server (branch hongweimeng/gguf-prompt-logprobs, -np 4 -c 131072), measured 2026-08-06. The reference columns were measured with vLLM (--tasks leaderboard, 2026-07-22).

On both the generative and the multiple-choice tasks this GGUF tracks the BF16 baseline closely, indicating faithful quantization export.

¹ Reference only, not measured on this GGUF file (BF16 and vLLM numbers from --tasks leaderboard, 2026-07-22). UInt4-GS128 is a groupsize=128 sibling model, different from this groupsize=32 GGUF; shown for context. ² Multiple-choice / loglikelihood task, measured on this GGUF via the patched llama-server (branch hongweimeng/gguf-prompt-logprobs) + lm-eval gguf backend. BBH/GPQA/MuSR measured 2026-08-06; MMLU-Pro completed 2026-08-08 (~114k loglikelihood requests, 12,032 samples, memory-bandwidth bound at ~0.6 it/s).