lackonendes/PAW-27B-X3.1-GGUF

PAW-27B-X3.1

A 3.5 bpw x3 trellis quantization of `Qwen/Qwen3.8-27B` that has lower code-prediction error than our earlier PAW-27B-X3 3.5-bit file, in the same size class (11.50 GiB). It runs on one 24 GB GPU.

What is new compared to PAW-27B-X3 3.5-bit

Two changes: a new codec rate (which needed a new decode kernel) and a better encoder. The decode format and kernels for K3 and K4 are unchanged.

1. 1.Fractional 3.5-bit codec (K3.5). The earlier file mixes 3-bit (K3) and 4-bit (K4) tensors. Here 395 of the 400 quantized matrices are stored at a true 3.5 bits per weight and 5 are stored at K3. The format is the exllamav3 fractional trellis (KA = 3, mask 0xAAAA). This needed a new decode kernel in llama-paw (4b936837b). The existing K1-K4 kernels are byte-for-byte the same machine code: cuobjdump -sass shows 86 of 86 existing kernels identical, plus 9 new ones.
2. 2.Newer encoder. The weights are quantized with exllamav3 v1.5.1.

Quality: KL vs the Q8_0 parent

Full-vocabulary KL of the model's logits against Q8_0 logits, 64 chunks of 2048 tokens, two sets: chatcode (chat-framed code, the primary) and rawcode (unframed code, the guard). All rows below come from the same binary, paired per chunk, bootstrap with 10,000 resamples (seed 20260911). Lower is better.

Both intervals are above zero, so this is a measured improvement over the earlier 3.5-bit file, not noise.

We also compared against rB, an unpublished intermediate build with the same size as the earlier file. X3.1 is 11.4% lower on chatcode (CI [0.00023, 0.00530], barely above zero) and 4.0% lower on rawcode (CI crosses zero, so that one is not a detected difference).

KL measures how close the logits are to Q8. It is not an accuracy score.

Benchmarks

Benchmarks for X3.1 use a thinking ladder, run greedy on one GPU:

• —P1: no thinking, 8,192-token cap, every item.
• —P2: no thinking, 32,768 cap, only P1-wrong items that hit the cap.
• —P3: thinking (xhigh), 32,768 cap, every item still wrong.
• —P4 / P5 / P6: thinking at 65,536 / 131,072 / 253,952, only items cut off by the previous cap.

The reported "final" score is the union over phases: an item counts if any phase got it right. That is an optimistic score and not the same as a single fixed-setting run.

Retention against the published BF16 model

Baseline: the "Base" column of the ThinkingCap-Qwen3.8-27B card. Base is Qwen/Qwen3.8-27B in BF16, the model PAW quantizes. (The card's "Ours" column is a different model, ThinkingCap.) We did not run BF16 ourselves. Retention = X3.1 final (union) score divided by that published mean.

Read this as a rough guide, not a like-for-like claim. The protocols differ:

• —Baseline: a mean over many independent seeds (32 on AIME 2026, 16 on GPQA-Diamond and IFBench, 8 on LiveCodeBench), sampled at temperature 1.0, topp 0.95, topk 20, with thinking at xhigh and a 253,952-token cap, on the full sets (MMLU-Pro 12,032 questions, IFBench 300, LiveCodeBench v6 175).
• —X3.1: one greedy run, the union over the ladder phases (an item counts if any phase got it right, which is optimistic), on subsets (MMLU-Pro 500, IFBench 150, LiveCodeBench 50; AIME 2026 and GPQA-Diamond are the full sets).
• —GPQA-Diamond: 11 of the 198 questions were cut off at the 32,768-token cap in phase 3 and were not run at the 65,536-token phase 4, to keep the benchmark schedule; they count as wrong here, so 179/198 is a floor for the ladder score (at most 190/198). They will be run later.
• —MMLU-Pro: phase 1 (no thinking, 8,192 cap) got 405/500 (81.0%), phase 2 added 4, and phase 3 (thinking, 32,768 cap) added 27. 20 items hit the 32,768-token cap in phase 3 and were still wrong; to keep the benchmark schedule they were not run at the longer phases 4+ and count as wrong, so 436/500 is a floor for the ladder score (at most 456/500). They will be run later. The retention figure is above 100% only because of the protocol differences above (greedy union on a 500-item subset vs a sampled mean on all 12,032 questions); it is not evidence that X3.1 beats BF16.
• —IFBench: phase 1 (no thinking, 8,192 cap) got 54/150 (36.0%), phase 2 added 0, phase 3 (thinking, 32,768 cap) added 62, and phase 4 (thinking, 65,536 cap) added 1, for 117/150 (78.0%) by the strict checker (118/150 by the loose one). 6 items hit the 65,536-token cap in phase 4 and were still wrong; to keep the benchmark schedule they were not run at the longer phases 5+ and count as wrong, so 117/150 is a floor for the ladder score (at most 123/150). They will be run later. The baseline card does not say whether its 79.75 is strict or loose. Also not like-for-like: 150-item greedy subset here vs a 300-item sampled mean in the baseline.
• —With 30 items on AIME 2026, one item is 3.3 points, so its 98.5% has wide error bars.
• —The union score is higher than what you get at any single fixed setting.

The earlier 3.5-bit file's own results (from the PAW-27B-X3 card, no thinking, different protocols) are: MMLU-Pro 67.4%, HumanEval 97.0% / HumanEval+ 92.1%, MBPP+ 78.6%, GSM8K 94%. We have not yet run those on X3.1, so we make no accuracy claim against them.

Files

Integrity:

4acefd28b247259cd25367ad7915ec6142b5f7a966d69b71599802fa63352599  PAW-27B-X3.1.gguf

Serving

Build llama-paw from main (commit 4b936837b or newer) with CUDA. This is the exact server line used for the benchmarks above (one RTX 3090, 24 GB):

export GGML_PAW_X3_GEMV=2 GGML_PAW_MMQ_HEAD=1 GGML_PAW_GREEDY_IDS=1 GGML_PAW_DQ4=0

./build/bin/llama-server -m PAW-27B-X3.1.gguf \
  -md Qwen3.8-27B-DFlash2-Q2_K.gguf \
  --spec-type draft-dflash --spec-draft-n-max 5 \
  -fa on -ctk q4_0 -ctv q4_0 \
  -c 262144 -ub 1024 -b 8192 -ubd 256 \
  -ngl 99 -np 1 --no-warmup \
  --reasoning off --reasoning-effort xhigh --reasoning-budget -1 \
  --host 127.0.0.1 --port 8080

• —Use `-ub 1024` at 262k context. K3.5 prefill does not fit at -ub 2048 there.
• —Thinking is chosen per request with chat_template_kwargs.enable_thinking (true or false). --reasoning-budget -1 means no thinking budget.
• —The K3.5 matrices only have the GEMV path (up to 8 tokens) and a reconstruct-then-cuBLAS path for larger batches. The tensor-core GEMM used by K3 and K4 for mid-size batches is not written for K3.5 yet, so prompt-processing speed on this file is not tuned. We have not measured it.

Limitations

• —Speed is unmeasured. Do not assume it matches the earlier 3.5-bit file.
• —LiveCodeBench is running, so its row is unfinished. Quality claims here rest on the KL measurement, AIME 2026, GPQA-Diamond (11 items still to be retried) and MMLU-Pro (20 items still to be retried) and IFBench (6 items still to be retried).
• —KL was measured on code text only (chat-framed and raw). Nothing is measured for other domains.
• —This is a code-focused build; general-domain quality is not measured.
• —One AIME item ends thinking with no answer; the ladder's fixed retry rule skips it, so the final AIME number depends on how that case is counted.

License

Apache 2.0, inherited from the base model Qwen/Qwen3.8-27B.

Credits and attribution

• —ExLlamaV3 by turboderp: the trellis codec design, and the v1.5.1 fractional-K trellis this release uses.
• —Earlier PAW-27B-X3 release and credits: PAW-27B-X3-GGUF.
• —Base model Qwen3.8-27B by Qwen.