kavenmartinez/Qwen3.6-27B-IQ2_XXS-webgpu-GGUF

Qwen3.6-27B — IQ2_XXS (WebGPU, runs headed on 12 GB cards)

A 2.06-bit-class (IQ2_XXS) imatrix quant of Qwen3.6-27B built to fit the full 27B model headed in the browser (i.e. while the same GPU is driving your desktop display) on 12 GB cards — validated on the RX 6700 XT and applicable to the RTX 3060 12 GB.

• —File: Qwen3.6-27B-IQ2_XXS-mtp.gguf
• —Size on disk: 8.88 GB (8462 MiB, 2.60 BPW)
• —Effective GPU weights: ~8.17 GB in the Artifex WebGPU engine (the MTP / next-token-prediction block is stripped on load — see notes)
• —Architecture: qwen35 hybrid (Gated DeltaNet + softmax attention layers)
• —Calibration: Unsloth imatrix (unsloth_calibration_Qwen3.6-27B.txt, 496 entries over 77 chunks)

Why this quant exists

The official Unsloth 2-bit builds are excellent but use the UD (dynamic) recipe, which (a) lands at ~9.4–9.6 GB and (b) can promote sensitive tensors to IQ types that some lightweight inference engines don't decode. This file is a flat-ish stock-imatrix IQ2_XXS — smaller (8.88 GB), and it sticks to a quant type set that the Artifex WebGPU engine decodes natively. That ~0.8 GB headroom is the difference between loads headed and device-lost on a 12 GB card whose compositor is also eating VRAM.

If you have VRAM to spare or run headless, prefer the official `unsloth/Qwen3.6-27B-MTP-GGUF` UD builds for slightly better quality.

Measured performance (greedy / deterministic)

Decode is GPU-bound (the IQ2_XXS matmuls are ~82% of each forward pass), so expect comparable numbers on the RTX 3060 12 GB.

Quality

This is a stock-imatrix IQ2XXS, not Unsloth Dynamic, so it gives up a little quality versus the UD-IQ2XXS for ~0.8 GB less footprint. In greedy testing the output is fully coherent and converges with the llama.cpp reference run — no 2-bit "word salad."

Perplexity (wikitext-2-raw test, n_ctx=512, 40 chunks, llama.cpp):

The 2-bit quant costs +2.31 PPL (~39%) versus the full-precision base — a real but non-catastrophic loss. The model stays coherent; it is not degraded into incoherence. (PPL at n_ctx=512 runs higher than the commonly-quoted 4096-context numbers; use the same settings if you compare.)

Running it

In the Artifex WebGPU engine (the intended target)

Get the engine from Artifex_Web and run it (see that repo's README for build/serve steps), then:

1. 1.Open the WebGPU app in a WebGPU-capable browser (Chrome/Edge).
2. 2.Enter this repo id in the model box: kavenmartinez/Qwen3.6-27B-IQ2_XXS-webgpu-GGUF
3. 3.Select your GPU adapter, load, and chat.

AMD cards (RX 6700 XT) — required step: the engine's auto VRAM detection is NVIDIA-only, so AMD cards fall back to a conservative default that's too small for a 27B and the load will fail. Set a manual budget on the app's origin before loading, via the browser devtools console:

localStorage.setItem('vramBudgetGB', '11.8'); // 12 GB card, ~1 GB left for the compositor

NVIDIA cards (RTX 3060 12 GB): auto-detection works; no override needed. If you still hit an OOM headed, set vramBudgetGB to 11.0–11.5.

In llama.cpp / llama-server

The file keeps its MTP block (blk.64, q8_0), so llama.cpp users get the multi-token-prediction speculative speedup for free:

llama-server -m Qwen3.6-27B-IQ2_XXS-mtp.gguf -ngl 99 -c 4096

On a 12 GB card you may need partial offload (-ngl < 99) depending on context length and KV cache size.

Notes & internals

• —MTP block: the GGUF contains the next-token-prediction block at q8_0 (~590 MB). The Artifex WebGPU engine strips it on load (it uses an n-gram drafter, not MTP), which is why effective GPU weights are ~8.17 GB rather than the 8.88 GB disk size. Kept in the file so llama.cpp can use it.
• —Quant mix: predominantly IQ2XXS, with embeddings/output and a handful of sensitive tensors kept at higher precision (Q4K / Q5K / Q2K), which is why the BPW is 2.60 rather than a pure 2.06.
• —Tokenizer: include tokenizer.json and tokenizer_config.json in this repo — the WebGPU engine loads them directly via AutoTokenizer.from_pretrained for remote repos, so the model is self-contained.

Credits & license

• —Base model: Qwen3.6-27B by the Qwen team (Apache-2.0).
• —Calibration imatrix: Unsloth (unsloth_calibration_Qwen3.6-27B.txt).
• —Quantization: stock llama-quantize IQ2_XXS with the Unsloth imatrix.

Released under Apache-2.0, inheriting the base model's license.