cyburn/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-PrismaQuant-8bit-vllm

Qwen3.6-35B-A3B — Claude 4.7 Opus Reasoning Distilled — PrismaQuant 8 bpp

  

Mixed-precision quantization of Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled produced by **PrismaQuant** — a per-Linear sensitivity-driven allocator that chooses each Linear module's format individually under a total-bit budget.

The source model is a reasoning distillation of Qwen3.6-35B-A3B trained on Claude 4.7 Opus chain-of-thought traces. It retains the full multimodal (vision + text) and MTP speculative-decoding head of the base architecture, with added reasoning capability. Enable thinking mode at inference time (see Serving below).

Why "every layer refracts into a different format": a naive uniform quantization either leaves disk on the table (keeping everything BF16 "to be safe") or loses quality (quantizing sensitive layers to 4-bit). PrismaQuant measures the actual Fisher-weighted MSE for every (Linear, format) pair and runs a multi-choice knapsack under a total-bit budget, so every bit lives where it buys the most likelihood.

At a glance

Precision mix

This checkpoint uses three precisions, selected per-Linear by the allocator from measured sensitivity — not chosen uniformly. The visual encoder is kept entirely at BF16 (--visual-format=BF16), preserving full visual fidelity at the cost of a small disk overhead vs a quantized visual tower.

Allocator result (text body, 401 Linears):

Export recipe (all modules, 511 entries including MTP + visual passthrough):

The allocator couples gate_up_proj / down_proj siblings so they share one per-tensor global scale (vLLM's FusedMoE requires this), and fused attention siblings (q_proj/k_proj/v_proj) share one per-tensor global scale so the packed qkv_proj loads without the "accuracy mismatch" warning. 112 fused-sibling groups were resolved during allocation.

61 Linears were ineligible for MXFP8_E4M3 due to kernel shape constraints and were restricted to the NVFP4 / BF16 choice.

Activation-aware passes applied during export

On every NVFP4 weight the exporter runs, in order:

1. 1.GPTQ-OBS one-shot rounding — block-wise error propagation along the group-quant structure using the calibration Hessian. Closed-form, not iterative. Handles cross-column activation coupling.
2. 2.Closed-form per-group scale sweep — for each 16-weight NVFP4 group, enumerate grid=32 candidate scales spanning [0.5·s₀, 1.5·s₀], round each weight to its nearest codebook neighbor at every candidate scale, pick the (scale, rounding-set) configuration minimizing activation-weighted per-group MSE sum_j a_j² · (w_orig,j - w_q,j)². Improve-or-keep gate against the post-GPTQ weight.

Measured per-Linear output-MSE vs RTN baseline (geomean):

AWQ's γ-fold is not applied. On NVFP4's 16-channel groups, AWQ's per-channel rescaling pushes mixed-scale values into the same group and inflates per-group quant noise rather than reducing it.

Pareto curve (body allocation)

The allocator sweeps a range of bit targets and reports predicted Δloss (Fisher-weighted MSE proxy). Target 8.0 bpp was chosen; the allocator suggested 4.5 bpp as the knee. The 8.0 target falls between the 7.0 and 8.25 table rows (achieved=0.695, NVFP4=56, MXFP8=29, BF16=193).

Which layers are quantized

Text body (DeltaNet linear-attention + dense MoE, 40 layers)

• —Full attention Linears (q_proj, k_proj, v_proj, o_proj): mixed NVFP4 / MXFP8 / BF16 per-Linear by sensitivity
• —DeltaNet linear-attention Linears (in_proj_qkv, in_proj_z, in_proj_a, in_proj_b, out_proj): same — note in_proj_a / in_proj_b variants were ineligible for MXFP8 due to shape constraints and resolved to NVFP4 or BF16
• —MoE experts (gate_up_proj, down_proj): mixed per-expert precision — 48 per-expert NVFP4 + 32 per-expert BF16 = 80 total per-expert entries; the allocator assigned BF16 to the highest-sensitivity experts and NVFP4 to the rest. Joint per-tensor scale across each gate_up pair satisfies vLLM's FusedMoE requirement.
• —Shared expert MLP: same per-Linear policy
• —Router (mlp.gate): always BF16 (tiny, sensitive)

Multi-token-prediction (MTP) head

Speculative-decoding head (1 layer) + its own MoE block quantized under the same per-Linear policy (3 089 MTP tensors merged at export). --speculative-config method=mtp drafts at the same precision as the body.

Visual encoder (110 Linears — BF16)

All 110 visual Linears are kept at BF16 via --visual-format=BF16. This is a deliberate conservative choice — the visual tower is left at full precision rather than running sensitivity-driven allocation on it. Disk overhead vs a quantized visual tower is small and visual quality is fully preserved.

Passthrough (unquantized)

• —lm_head — kept at BF16. vLLM's ParallelLMHead only accepts a single weight parameter; the compressed-tensors runtime rejects a compressed lmhead with `KeyError: lmhead.inputglobalscale`. This is a vLLM runtime limitation, not a PrismaQuant design decision. The allocator would have chosen a lower precision here.
• —RMSNorm weights (all layers + MTP + visual)
• —All biases
• —embed_tokens
• —model.visual.pos_embed (Parameter/Embedding, not an nn.Linear)

Serving (vLLM only)

This artifact is only runnable via vLLM's stock compressed-tensors support. vLLM 0.11+ is required.

vllm serve cyburn/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-PrismaQuant-8bit-vllm \
    --trust-remote-code \
    --max-model-len 32768 \
    --gpu-memory-utilization 0.90 \
    --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Thinking / reasoning mode

This model was distilled from Claude 4.7 Opus chain-of-thought traces. Enable reasoning at inference time via the chat template:

messages = [{"role": "user", "content": "..."}]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True,   # enables <think>...</think> CoT blocks
)

Or pass /think at the start of your prompt in interactive use. The model works without thinking mode enabled — it will respond directly — but reasoning quality is higher with it on.

• —FlashInfer NVFP4 attention is picked up automatically; set VLLM_USE_FLASHINFER_NVFP4=1 to make the preference explicit.
• —MTP speculative decoding at n=3 is the measured optimum for this family (n=2 leaves ~10 % tok/s on the table, n=4 regresses).
• —Visual inputs work via vLLM's standard image-text-to-text chat API — no special flags.

Reproducing this artifact

Full pipeline is in the PrismaQuant repo:

1. 1.Sensitivity probe — streaming per-shard empirical-Fisher trace (diagonal) across body + MTP Linears.
2. 2.Per-(Linear, format) cost measurement — for each Linear and each candidate format, the per-group RTN error weighted by cached input activations.
3. 3.Multi-choice knapsack allocator — picks one format per Linear minimizing total predicted Δloss under the bit budget. Target 8.0 bpp.
4. 4.Export — streams each body / MTP shard, applies GPTQ + activation-weighted scale sweep to its NVFP4 entries, writes the compressed-tensors format. Visual encoder stamped uniformly at BF16 via --visual-format=BF16.

Known issues / limitations

• —vLLM only at serve time. No transformers-runtime path for this precision mix today.
• —lm_head stays BF16 because vLLM's ParallelLMHead does not register the NVFP4/MXFP8 compressed-tensors schemes. Costs a small amount of disk vs full quantization.
• —Visual encoder is BF16. If disk size is the primary constraint and visual quality is not critical, a variant with sensitivity-driven visual quantization (as in rdtand/Qwen3.6-35B-A3B-PrismaQuant-4.75bit-vllm) could reduce the visual tower to ~22 % of its BF16 size.
• —MTP n=4 regresses on this family. Stick to n=3 unless you verify against the draft-head acceptance-rate trace.

Links

• —PrismaQuant: github.com/RobTand/prismaquant
• —Base model: Qwen/Qwen3.6-35B-A3B
• —4.75 bpp sibling (Claude 4.7 Opus distilled): cyburn/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-PrismaQuant-4.75bit-vllm
• —Non-distilled sibling: rdtand/Qwen3.6-35B-A3B-PrismaQuant-4.75bit-vllm

Citation

@software{prismaquant2026,
  title        = {PrismaQuant: per-Linear sensitivity-driven mixed-precision
                  quantization for LLMs},
  author       = {Tand, Rob},
  year         = 2026,
  url          = {https://github.com/RobTand/prismaquant},
}