a1700878328/qwen3-asr-1.7b-onnx-int4

Qwen3-ASR-1.7B ONNX deployment build

This directory contains a locally exported deployment build of Qwen/Qwen3-ASR-1.7B.

Format

• —ONNX opset 18
• —native FP16 audio encoder with FP32 inputs and outputs
• —INT4 RTN decoder, block size 64, ONNX Runtime accuracy level 4
• —FP16 token embedding table
• —shared external decoder weights
• —model directory size: about 3.5 GB

This is a split native ONNX Runtime pipeline. It is not a sherpa-onnx model and cannot be dropped into a sherpa-onnx runner without an adapter.

Provenance

• —Base model: Qwen/Qwen3-ASR-1.7B, Apache-2.0
• —Exporter: andrewleech/qwen3-asr-onnx
• —Exporter commit: 4aee2ae0c355c7bc33bd5c09befec9e9eabd4b50
• —Python 3.13
• —PyTorch 2.11.0+cu128
• —ONNX Runtime GPU 1.23.2

The official model was downloaded and verified locally before export. The encoder, decoder-init, and decoder-step graphs all passed ONNX Runtime loading checks. CUDA validation rejects silent CPU fallback.

Local validation

Run from the exporter repository:

$env:PYTHONUTF8 = "1"
uv sync
uv run python validate.py `
  --model models/Qwen3-ASR-1.7B `
  --onnx-dir release/qwen3-asr-1.7b-onnx-int4 `
  --audio C:\path\to\test.wav `
  --provider cuda `
  --max-tokens 1024

The validation command also loads the original PyTorch model for comparison, so it is slower and requires the original weights. Production inference needs only the ONNX model files, tokenizer, audio preprocessing, and ONNX Runtime.

Measured results

Measured on an RTX 5070 Ti 16 GB:

Normalized character error rate against the two supplied transcripts was 3.21% and 5.14%, or 4.20% aggregate. Punctuation, spaces, and symbol differences were excluded. A company RTX 4090 should be faster, but it must be benchmarked on that machine before setting concurrency limits.

Reproduction outline

uv run python export.py `
  --model models/Qwen3-ASR-1.7B `
  --output output/qwen3-asr-1.7b-fp32 `
  --device cpu `
  --dtype fp32 `
  --opset 18 `
  --no-share-weights

uv run python share_weights.py output/qwen3-asr-1.7b-fp32 --verify

uv run python quantize_nbits.py `
  --input output/qwen3-asr-1.7b-fp32 `
  --output output/qwen3-asr-1.7b-fp32 `
  --bits 4 `
  --block-size 64 `
  --accuracy-level 4 `
  --algo rtn

uv run python share_weights.py `
  output/qwen3-asr-1.7b-fp32 `
  --suffix int4 `
  --verify

uv run python export_encoder_native_fp16.py `
  --model models/Qwen3-ASR-1.7B `
  --output output/qwen3-asr-1.7b-fp32/encoder.int4.onnx `
  --opset 18 `
  --verify

uv run python convert_embed_fp16.py `
  --model-dir output/qwen3-asr-1.7b-fp32

The release directory was assembled from the native FP16 encoder, shared INT4 decoder pair, shared INT4 external weights, FP16 embeddings, tokenizer, and configuration files.