qualcomm-ai-hub-community/whisper-small-multilingual-qcs6490-qnn-int8

Whisper-Small (multilingual) — QCS6490 · QNN context binary · full INT8

Companion to the whisper-base build — the higher-accuracy Whisper-Small variant compiled and profiled for the Qualcomm Dragonwing QCS6490 (Hexagon HTP), the SoC on Rubik Pi 3 / RB3 Gen 2 / Radxa Dragon Q6A class devices. Fills the gap discussed in qualcomm/ai-hub-models#281.

Measured on device (Qualcomm AI Hub hosted Dragonwing RB3 Gen 2 Vision Kit)

≈ 1.0 s for a 30-token transcript; the smaller whisper-base build (also in this org) runs ≈ 3× faster where its accuracy suffices.

Why full INT8 (read this before recompiling anything for QCS6490)

Measured constraint map for this target (all jobs on AI Hub, 2026-08-02):

1. 1.Float graphs are rejected at validation — "floating-point type is not supported by the targeted device". This target requires fully-quantized graphs including I/O.
2. 2.W8A16 fails context-binary conversion (exit code 14) — reproduced on whisper-small and whisper-small, on both qnn_context_binary and precompiled_qnn_onnx. 16-bit activations appear unsupported on this HTP generation (jobs jgjq11k85, jgznwwv6g, jgnkxxxmg, jpymnnn4p).
3. 3.Full INT8 (`--quantize_full_type int8 --quantize_io`) works — these binaries.

No local AIMET is needed at any point (AIMET-ONNX is unavailable on macOS/aarch64 — the original blocker in #281): quantization here happens server-side at compile; for accuracy-critical deployments, use AI Hub's hosted quantize jobs with calibration audio, then compile the same way.

Reproduce

# Python >= 3.10; pip install "qai-hub-models[whisper-small]"; qai-hub configure --api_token <yours>
python -m qai_hub_models... # or drive qai_hub directly:

import qai_hub as hub
from qai_hub_models.models._shared.hf_whisper.model import HfWhisperEncoder, HfWhisperDecoder

device = hub.Device("Dragonwing RB3 Gen 2 Vision Kit")
for name, cls in (("encoder", HfWhisperEncoder), ("decoder", HfWhisperDecoder)):
    component = cls.from_pretrained(hf_whisper_version="openai/whisper-small").eval()
    spec = component.get_input_spec()
    job = hub.submit_compile_job(
        model=component.convert_to_torchscript(spec), device=device, input_specs=spec,
        options="--target_runtime qnn_context_binary --quantize_full_type int8 --quantize_io",
        name=f"whisper-small-{name}-int8")

Note: blind (calibration-free) INT8 trades some accuracy for availability — validate WER on your domain before production use. The same recipe works for any HF Whisper fine-tune (e.g., vinai/PhoWhisper-small for Vietnamese) by changing hf_whisper_version.

I/O

Quantized I/O per --quantize_io — exact tensor specs are on each job page above. Runs via Qualcomm AI Engine Direct (QNN) context-binary loading, or ONNX Runtime QNN EP with context binary support.

Intended use & limitations

• —Intended use: offline multilingual speech recognition on QCS6490-class edge devices (IoT gateways, dev boards, embedded products); development reference for compiling Whisper-family models to this target.
• —Accuracy: quantization here is calibration-free (server-side --quantize_full_type int8). WER against the FP32 baseline has not been measured — validate on your domain before production use; for accuracy-critical deployments run an AI Hub hosted quantize job with calibration audio first.
• —Scope: these binaries target the QCS6490/Qualcomm Linux runtime; they will not load on other chipsets — recompile with the recipe above for your target.
• —Whisper's known behaviors (hallucination on silence/non-speech, fixed 30 s window) carry over; gate inputs with a VAD.

Attribution & license

Base model: openai/whisper-small (MIT). Compiled artifacts follow the base model's MIT license. Export wrappers: qualcomm/ai-hub-models (BSD-3-Clause, SHA-attention rewrite). Prepared by Phạm Văn Ngoan. Not an official Qualcomm release; community-contributed, moderated for structure only.