OpenVoiceOS/lite-whisper-large-v3-turbo-onnx

lite-whisper-large-v3-turbo — ONNX (fp32 only)

ONNX export of efficient-speech/lite-whisper-large-v3-turbo for onnx-asr (standard whisper model type — works with stock onnx-asr, no patches needed).

Export notes

This model uses a custom LiteWhisperForConditionalGeneration architecture (model_type: lite-whisper): the encoder's linear layers are replaced with low-rank factorized (LinearLowRank) layers for structural compression. optimum's exporter does not recognize the custom model_type (it isn't registered in transformers' CONFIG_MAPPING_NAMES, so TasksManager can't resolve a model class for it even with trust_remote_code=True and manual Auto*.register() calls — those populate the instance-level _LazyAutoMapping but TasksManager.get_model_class_for_task reads transformers' string-keyed CONFIG_MAPPING_NAMES instead). Worked around by loading the model directly (AutoModel.from_pretrained(..., trust_remote_code=True), matching the AutoModel entry in the repo's auto_map) and exporting via optimum's low-level onnx_export_from_model(model=..., ...), which skips TasksManager's model-type resolution. The underlying forward pass is structurally identical to WhisperForConditionalGeneration (only the encoder's nn.Linear layers are swapped for two-matmul low-rank layers), so the standard WhisperOnnxConfig traces correctly.

The source repo ships no tokenizer; vocab_size (51866) matches stock openai/whisper-large-v3-turbo, whose tokenizer is bundled here (same precedent as the WhisperLv3-FT export in OpenVoiceOS/inesc-id-whisperlv3-ft-ep-onnx).

int8 is not published. Standard onnxruntime dynamic quantization (quantize_dynamic, QInt8) of the encoder produces a broken model: decoding loops on a near-blank token for the full max_length, while the same decoder paired with the fp32 encoder works. Isolated by swapping fp32/int8 sub-models independently — the defect is confined to the int8 encoder, i.e. it's the low-rank factorized layers that don't survive this quantization recipe, not the decoder or the merge step (which is otherwise byte-identical to the working recipe used for the other Whisper exports in this collection). Shipping fp32 only rather than a broken int8 artifact.

Usage

import onnx_asr
model = onnx_asr.load_model("whisper", "path/to/this/repo")
print(model.recognize("audio_16khz.wav", language="en"))

Verified on FLEURS English and Portuguese clips: fp32 output matches the native transformers pipeline output exactly, for both languages. RTF on an AMD Ryzen 5 7600 (6C/12T CPU, shared/loaded host): ~0.32-0.44 (this compressed encoder is meaningfully faster than full large-v3, even before quantization).