amd-satre/Qwen3.6-35B-A3B-w_int4_a_bf16-gptq

Qwen3.6-35B-A3B-wint4a_bf16-gptq

Qwen/Qwen3.6-35B-A3B quantized with AMD Quark.

What is quantized

Quark's built-in qwen3_5_moe recipe quantizes the routed MoE experts only — 30,720 linears (40 layers x 256 experts x 3 projections). 462 modules are excluded and remain BF16:

lm_head, model.visual.* (vision tower), mtp.*, *mlp.gate, *shared_expert_gate*, *.linear_attn.*, *.self_attn.*, *.shared_expert.*

Because attention, the shared experts, the router and the vision tower all stay BF16, the size reduction is smaller than a whole-model quantization would give.

Reproduction

python examples/torch/language_modeling/llm_ptq/quantize_quark.py \
  --model_dir Qwen/Qwen3.6-35B-A3B \
  --device cuda --data_type bfloat16 --model_attn_implementation eager \
  --dataset pileval --num_calib_data 128 --seq_len 512 --batch_size 1 \
  --quant_scheme int4_wo_128 \
  --quant_algo gptq --quant_algo_config_file gptq <config.json> \
  --preserve_fused_experts \
  --model_export hf_format --export_weight_format real_quantized \
  --output_dir <out> --skip_evaluation

Notes

• —Quantized on a single AMD Instinct MI355X (gfx950) with Quark 0.13+50dac6e159a.
• —Calibration: pileval (mit-han-lab/pile-val-backup), 128 samples, seq_len=512.
• —The vision tower is not quantized, so this remains a full vision-language model.
• —GGUF export is not possible for this architecture: Quark's GGUF writer maps only MODEL_ARCH.LLAMA and supports asymmetric INT4 per-group-32 weight-only on Llama2/Llama3, while this model is Qwen3_5MoeForConditionalGeneration. Safetensors is published instead.
• —No accuracy evaluation has been run on this checkpoint. Benchmark before use.