SocialLocalMobile/Qwen3.6-35B-A3B-HQQ-INT4

Qwen3.6-35B-A3B Sensitivity-Aware Mixed Precision (HQQ)

Mixed-precision quantized version of Qwen/Qwen3.6-35B-A3B for ExecuTorch CUDA export.

Quantization Details

Qwen 3.6 does not have quantization-aware training, so uniform INT4 quantization destroys GatedDeltaNet recurrent state and MoE expert outputs. This checkpoint uses sensitivity-aware mixed precision based on per-layer error profiling and GGUF Q4KM analysis.

• —Expert quantization: Uses HQQ (Half-Quadratic Quantization) scale-only optimization with iterative least-squares scale refinement.
• —INT8 layers: Standard symmetric weight-only quantization via torchao.
• —bf16 layers: Kept at full precision — these are small tensors where quantization causes catastrophic errors in the GatedDeltaNet recurrent path.

File Format

• —model.safetensors — Quantized weights. Tensor subclasses (IntxUnpackedToInt8Tensor) are flattened into plain inner tensors with .__qdata / .__scale / .__zero_point suffixes. Expert weights are stored as packed INT4 buffers (w1, w1_scale, w2, w2_scale). Reconstruction metadata is in the safetensors header under "quantization".
• —config.json — Model architecture configuration.
• —tokenizer.json, tokenizer_config.json, merges.txt, vocab.json — Tokenizer files for runtime.

Prerequisites

• —ExecuTorch installed from source (see building from source)
• —safetensors (pip install safetensors)
• —NVIDIA GPU with CUDA toolkit

How to Use

Eager Inference (Python)

cd executorch/examples/models/qwen3_5_moe
python inference.py \
    --prequantized /path/to/Qwen3.6-35B-A3B-HQQ-INT4 \
    --prompt "The capital of France is" \
    --max-new-tokens 128

Export to ExecuTorch (.pte)

cd executorch/examples/models/qwen3_5_moe
python export.py --prequantized /path/to/Qwen3.6-35B-A3B-HQQ-INT4

Build and Run (C++)

make qwen3_5_moe-cuda

cmake-out/examples/models/qwen3_5_moe/qwen3_5_moe_runner \
    --model_path exports/model.pte \
    --data_path exports/aoti_cuda_blob.ptd \
    --tokenizer_path /path/to/Qwen3.6-35B-A3B-HQQ-INT4/tokenizer.json \
    --prompt "The meaning of life is" \
    --max_new_tokens 128

How to Reproduce

cd executorch/examples/models/qwen3_5_moe

python quantize_and_save.py \
    --model-dir /path/to/Qwen3.6-35B-A3B \
    --sensitive \
    --hqq \
    --output Qwen3.6-35B-A3B-HQQ-INT4

Requires CUDA and ~70GB RAM (for loading the original bf16 model).

Base Model

• —Model: Qwen/Qwen3.6-35B-A3B
• —Architecture: 40-layer hybrid transformer, 256 routed experts (top-8), GatedDeltaNet + full attention
• —Parameters: 35B total, 3B active per token
• —License: Apache 2.0