hyperhuzaifa/Qwen3.6-35B-A3B-W4A16-AWQ

Qwen3.6-35B-A3B-W4A16-AWQ

A 4-bit (W4A16) quantization of Qwen/Qwen3.6-35B-A3B, produced with llm-compressor using AWQ activation-aware calibration, in compressed-tensors pack-quantized format.

Built to run a 35B MoE with long context on a single 24 GB consumer GPU. Multimodal (vision) and the MTP speculative-decoding head are both preserved.

~20.5 GB on disk, 19.06 GiB resident on GPU.

Quantization details

moe_calibrate_all_experts matters here: with top-8-of-256 routing, each expert would otherwise see only ~3% of calibration tokens.

Quantized to int4

All language-model Linear layers — including the ones many public MoE quants skip: the 256 routed experts per layer, the shared experts, the 30 gated-DeltaNet (linear-attention) projections (in_proj_qkv, in_proj_z, in_proj_b, in_proj_a), and the 10 full-attention blocks.

Deliberately kept bf16

Serving with vLLM

Requires vLLM ≥ 0.26.0 (architecture Qwen3_5MoeForConditionalGeneration).

⚠️ You must set --max-num-seqs

This is a hybrid architecture: 30 of 40 layers are gated DeltaNet ("Mamba"-style). vLLM allocates one recurrent cache block per decode sequence regardless of context length, and on a 24 GB card only ~46 fit. vLLM's default --max-num-seqs 256 makes CUDA graph capture abort:

ValueError: max_num_seqs (256) exceeds available Mamba cache blocks (46).

Pass --max-num-seqs 46 or lower on a single 24 GB GPU. This caps concurrency independently of KV tokens.

Single 24 GB GPU

vllm serve hyperhuzaifa/Qwen3.6-35B-A3B-W4A16-AWQ \
  --max-model-len 32768 \
  --max-num-seqs 16 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.94

Two GPUs (recommended for long context)

vllm serve hyperhuzaifa/Qwen3.6-35B-A3B-W4A16-AWQ \
  --tensor-parallel-size 2 \
  --max-model-len 262144 \
  --max-num-seqs 32 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.90 \
  --disable-custom-all-reduce

Optional: MTP speculative decoding

The draft head is included. Enable with:

--speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":1}'

mtp_num_hidden_layers=1, so 1 speculative token is the maximum.

The MTP weights cost nothing until you enable this flag — vLLM only instantiates the module under --speculative-config, so otherwise they sit on disk. Enabling it makes the draft head resident (+1.57 GiB) and raises per-token KV cost.

⚠️ Do not enable MTP on a single 24 GB card. The draft head consumes ~92% of the KV budget and even 8K context then fails to allocate:

Measured MTP behaviour on 2 × 24 GB over PCIe (no NVLink): the draft head is very good — mean acceptance length 1.90 / 2.0, ~90% acceptance — but end-to-end throughput still drops, because each draft forward adds an all-reduce and the draft is a full 256-expert MoE block:

Raising --max-num-batched-tokens (which vLLM itself suggests) made it worse. On this hardware, serve without speculative decoding. On an NVLink pair the 1.90 acceptance length would likely convert into a real speedup — the bottleneck is interconnect, not the draft head.

Measured performance

Measured on RTX 3090 / RTX 4090 (24 GB each), fp8 KV cache, vLLM 0.26.0.

TP=2 gives a KV pool of 1,965,063 tokens. On a single card, full 262K context is not reachable — it needs 2.58 GiB of KV against ~1.71 GiB available.

Context and throughput trade off sharply on one card: --enforce-eager buys ~17K tokens of context but costs ~8× decode speed.

Known issues and gotchas

`lm_head` is intentionally NOT quantized. vLLM builds it as a plain ParallelLMHead for this architecture and requires lm_head.weight. A packed head fails at load with:

ValueError: There is no module or parameter named 'lm_head.weight_packed'

Load with the multimodal class. AutoModelForCausalLM resolves to Qwen3_5MoeForCausalLM (text-only) and will silently drop the vision tower and MTP head. vLLM does not register that architecture at all. Use Qwen3_5MoeForConditionalGeneration.

Vision is preserved but was not calibrated. The tower is bf16 and untouched, so image quality should match the base model, but it received no activation-aware scaling.

Not evaluated on benchmarks. Coherence was verified by generation (code synthesis, instruction following, long-context recall) rather than by lm-eval-harness or similar. No perplexity or task-accuracy numbers are published here — treat quality claims as unverified.

License

Apache 2.0, inherited from the base model. See the base model's LICENSE.