LayerNorm/Qwen3-ASR-0.6B-hf-hqq-int4

Qwen3-ASR-0.6B-HQQ-INT4

This repository contains an HQQ (Half-Quadratic Quantization) 4-bit version of `Qwen/Qwen3-ASR-0.6B-hf`,quantized for deployment on memory-constrained GPUs.

THIS IS NOT THE OFFICAL RELEASE.

• —Base model: Qwen3-ASR-0.6B-hf
• —Scheme: HQQ-INT4 (nbits=4, group_size=64, axis=1, compute dtype bfloat16)
• —Excluded layers: lm_head
• —Disk size: ~650 MB (vs ~1503 MB for the bf16 checkpoint, ~57% smaller)

The model weights are stored as packed HQQ tensors (W_q plus the per-module quantization config, e.g. scale/zero/nbits). Modern transformers + hqq do not yet reconstruct HQQLinear layers straight from a checkpoint, so do not load this with `AutoModelForMultimodalLM.from_pretrained` directly — it silently drops the quantized weights. Use the bundled `scripts/infer_hqq_int4.py` (or the reproduction steps below) that rebuilds the HQQLinear modules from the saved tensors.

Usage

# runtime deps for loading/transcribing (this model only needs transformers + torch + hqq)
pip install transformers==5.15.1 torch==2.13.0 hqq==0.2.8.post1

# analysis dep (compare_baseline.py computes CER):
pip install jiwer

Transcribe a clip (recommended path — reconstructs the HQQ layers correctly):

import sys
sys.path.insert(0, "scripts")
from infer_hqq_int4 import reconstruct_model
import torch
from transformers import AutoProcessor

checkpoint_dir = "."   # this repo
processor = AutoProcessor.from_pretrained(checkpoint_dir)
model = reconstruct_model(checkpoint_dir)   # restores the HQQLinear layers
model.eval()

inputs = processor.apply_transcription_request(
    audio="https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_en.wav",
).to(model.device, model.dtype)

with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=256)
gen = out[:, inputs["input_ids"].shape[1]:]
print(processor.decode(gen, return_format="transcription_only")[0])

Evaluation vs. baseline (bf16)

Measured on an RTX 3060 12GB (CUDA 13.3, Ampere, compute 8.6) on a real Chinese corpus (AISHELL-1 short clips + a ~47 s long clip + an 8 dB-noise clip + the official demo clip, 10 samples total). CER = character error rate (CJK/latin alphanumerics only); latencies are the median of 3 generate calls.

Takeaways

• —Nearly lossless accuracy. Across all samples the quantization-degradation CER vs. the bf16 model is ~0.006 (a single character difference on the noisy clip). Absolute accuracy on these clean Chinese clips is actually slightly better than bf16 (CER 0.0157 vs 0.0231).
• —~57% smaller and ~57% less GPU memory — the main win. Latency grows (~1.8–2×) on consumer Ampere GPUs, a typical quantized-decode trade-off.

Reproduce the numbers with `scripts/compare_baseline.py` (needs the local Chinese test set described in scripts/compare_baseline.py).

Reproduction

# 1) Quantize the base model to HQQ-INT4 (default output = this repo root)
python scripts/quantize_hqq_int4.py --base Qwen/Qwen3-ASR-0.6B-hf

# 2) Verify it reloads and transcribes correctly
python scripts/infer_hqq_int4.py --audio <some.wav>

# 3) Compare against the bf16 baseline (Chinese test set)
python scripts/compare_baseline.py

Citation

@article{Qwen3-ASR,
  title={Qwen3-ASR Technical Report},
  author={Xian Shi, Xiong Wang, Zhifang Guo, Yongqi Wang, Pei Zhang, Xinyu Zhang, Zishan Guo,
          Hongkun Hao, Yu Xi, Baosong Yang, Jin Xu, Jingren Zhou, Junyang Lin},
  journal={arXiv preprint arXiv:2601.21337},
  year={2026}
}