chtisgit/Qwen3.6-35B-A3B-2Bit-GSQ-hybrid-GGUF

Qwen3.6-35B-A3B 2-bit GSQ — GGUF

Overview

This is a GGUF conversion of `ISTA-DASLab/Qwen3.6-35B-A3B-2Bit-GSQ`. It is essentially the DASLab GSQ checkpoint repackaged into stock llama.cpp types, with the small conversion errors documented below. It is not a new quantization, fine-tune, or model release. It is distributed as a text-model GGUF plus a separate BF16/F32 vision-projector GGUF.

The routed experts use Q20 (source: 2-bit GSQ), while attention, linear attention, shared-expert, and LM-head weights use Q80 (source: INT8). Remaining text-model tensors use BF16 or F32 (source: BF16) as required by llama.cpp's canonical QWEN35MOE layout. The companion projector retains the source BF16 vision tower as BF16 or exact F32 promotions. MTP weights remain omitted.

Q2_0 is a recent llama.cpp type. Its type definition and CPU backend landed on 7 July 2026 in commit `bec4772`, first released as nightly build `b9913` on 8 July

1. 1.CUDA support followed on 30 July 2026 in commit `9b2a088`. Use a current llama.cpp build; older releases cannot load or run this model correctly.

The text model and projector were tested together with unmodified llama.cpp on CPU and with 20 of 41 text-model layers plus the projector offloaded to an RTX

1. 1.Both runs correctly read MEN WALK ON MOON from the supplied newspaper test image. These are load and short-generation checks, not benchmark or perplexity results.

Data-type mappings and their effects

GSQ 2-bit routed experts → Q2_0

Before applying their block scales, the two formats represent these integer sets:

GSQ 2-bit: {-2, -1, 0, 1}
Q2_0:      {-1,  0, 1, 2}

The source GSQ code c ∈ {0, 1, 2, 3} has integer value c - 2, which is multiplied by scale s. A stock Q2_0 code q ∈ {0, 1, 2, 3} has integer value q - 1, which is multiplied by scale d. Setting d = -s and reversing the code order makes the scaled values identical:

This directly gives the assignment-preserving conversion formula:

Q2_0 code  = 3 - c
Q2_0 scale = FP16(-s)

Each source group of 128 weights becomes two consecutive Q20 blocks of 64 that share the mapped scale. The 2-bit assignments are rearranged directly; the weights are not dequantized and requantized. The only numerical change is from converting each BF16 scale to the FP16 scale stored by Q20.

INT8 weights → Q8_0

The signed INT8 codes are copied directly into Q80 blocks. Each source group-of-128 scale is converted to FP16 and repeated across four Q80 blocks of 32. The ordinary Q8_0 quantizer is not used, so the codes—including -128 and +127—are preserved. Scale conversion is the only numerical change.

Across the full audited checkpoint, this direct Q8_0 representation had lower maximum error, mean absolute error, RMSE, and relative Frobenius error than dequantizing the same weights to BF16. It also reduced the finished text-only GGUF by 1,794,048,032 bytes compared with the BF16-fallback build.

Other tensors

Embeddings, norms, routing weights, biases, and other ordinary text tensors follow llama.cpp's canonical Qwen conversion and are stored as BF16 or F32. Expert aggregation and linear-attention permutations also follow the canonical QWEN35MOE layout.

BF16 vision tower → BF16/F32 mmproj

The source contains 333 vision tensors, all BF16. llama.cpp's stock Qwen3-VL converter stores the large vision-transformer and merger matrices as BF16 and promotes biases, normalization parameters, patch embeddings, and positional embeddings to F32. The temporal-size-2 Conv3D patch embedding becomes two Conv2D tensors, producing 334 GGUF tensors in total.

All 446,571,248 projector values were compared against the source. BF16 payloads are bit-exact and BF16-to-F32 promotion is exact, so this projector conversion has zero numerical error.

Accepted conversion errors

Stock Q20 and Q80 store their block scales as FP16, whereas the source checkpoint uses scales that are not always exactly representable in FP16. The following deviations were measured and explicitly accepted.

Q2_0 routed experts

• —1,890,976 scales with the previously identified BF16 bit pattern 0x018e underflow to FP16 zero. Each scale has magnitude 5.216256181773901e-38; setting it to zero produces a maximum weight error of 1.0432512363547802e-37 for this group.
• —15 additional scales incur FP16 subnormal rounding.
• —One additional scale of magnitude 2.200249582529068e-8 underflows to zero, producing a maximum weight error of 4.400499165058136e-8 for that group.
• —The maximum measured GSQ-to-Q2_0 weight error is 5.960464477539063e-8.

The direct mapping was retained rather than changing assignments to hide these errors.

Q8_0 INT8-derived weights

• —7,104 nonzero scales underflow to FP16 zero, all in seven linear-attention tensors. They produce 28,416 zero-scale Q8_0 blocks, cover 909,312 weight positions, and incur a maximum weight error of 1.6699515299478662e-6.
• —No scale overflow or additional scale-loss regime was found.

Despite those underflows, Q8_0 has lower whole-tensor maximum error, mean absolute error, RMSE, and relative Frobenius error than BF16 for each of the seven affected tensors.

Basic llama.cpp usage

llama-cli \
  --model Qwen3.6-35B-A3B-GSQ-hybrid.gguf \
  --mmproj mmproj-BF16.gguf \
  --image image.jpg \
  --ctx-size 4096 \
  --gpu-layers 20 \
  --prompt "Describe this image."

Omit --mmproj and --image for text-only use. Adjust context size and GPU offload for the available memory. For grounding-sensitive image tasks, llama.cpp recommends --image-min-tokens 1024. This conversion was validated against llama.cpp commit 911f6cdc8ab8a530b2bee09ee61471a6f3178eeb.

Attribution

The model weights and GSQ quantization are from `ISTA-DASLab/Qwen3.6-35B-A3B-2Bit-GSQ`, which is based on `Qwen/Qwen3.6-35B-A3B`. Please cite and credit the original model and GSQ authors when using this conversion.

License

This GGUF conversion is made available under the Apache License 2.0, just like the DASLab source checkpoint and its Qwen base model.