collectiveai/canary-1b-v2-onnx-split-int8

Canary-1B-v2 ONNX — split decode-loop + INT8 (encoder & decoder)

A modified ONNX export of `nvidia/canary-1b-v2` (CC-BY-4.0), built on top of `istupakov/canary-1b-v2-onnx` (also CC-BY-4.0). Two independent changes on top of that export:

1. 1.Split decode-loop (graph surgery): the original fused decoder-model.onnx recomputes 16 cross-attention key/value projections from scratch on every decode step (~144 times per 30 s window), even though they never change within a window. This repo splits that graph in two — xattn_kv.onnx (K/V, computed once per window) and decoder_step.onnx (the per-token loop, now takes K/V as inputs) — and verifies the split composes back to the original fused graph bit-for-bit (max output difference 0.0 across a held-out set). Zero transcription drift, pure speed fix.
2. 2.INT8 quantization, independently, on each half:
3. 3.decoder_step.dynamic_v1_quint8.onnx: dynamic quantization (onnxruntime.quantization.quantize_dynamic, QuantType.QUInt8, MatMul-only, no calibration data needed).
4. 4.encoder-model.static_qdq_v4_pct_excl.onnx (+ .onnx.data): static QDQ INT8 (Conv/MatMul/Gemm, per-channel, reduce_range=True), with percentile (99.999%) activation calibration over a 40-clip, 5-language calibration set, and 32 nodes measured and excluded via onnxruntime.quantization.qdq_loss_debug (late-layer convolutions and MatMuls whose per-tensor quantization SNR was a 6–22 dB outlier against a ~33 dB median — leaving those in fp32 is what actually closes the quality gap; percentile calibration alone was not enough).

This repository ships only the modified/quantized artifacts — the unmodified fp32 encoder-model.onnx, vocab.txt and config.json are included here too (byte-identical to the base repo) purely for convenience, so this directory is self-contained and you don't need to fetch two repos to use it.

Quality and speed (full 48-window mTEDx long-form validation, cpWER)

All numbers below are the normalized cpWER on the same held-out, real-world long-form Spanish talk (mTEDx, 48 non-overlapping 30 s windows, scored via meeteval's cpWER) — not a short clean-clip screen, which this project's own history has twice shown can diverge sharply (in both directions) from full-scale results.

Caveat on RTFx: the encoder-INT8 row's RTFx was measured in the same run as its own fp32 control (1.2112 → 1.2600, +4.0% relative) but on a different host-load condition than the split-decode baseline's RTFx 1.62 figure, so do not read those two RTFx numbers as directly comparable in absolute terms — only the relative (fp32-vs-INT8, same run) comparisons are apples-to-apples. The combined configuration (INT8 encoder and INT8 decoder together, one clean run) has not yet been measured — that's the natural next benchmark for anyone building on this.

Important: not a drop-in onnx_asr model

Because of the split decode-loop, this artifact set is not loadable by plain onnx_asr.load_model(...) the way the base repo is — it needs a decode loop that knows about the xattn_kv.onnx / decoder_step.onnx split and the two independent quantization selectors. Reference implementation: `onnx-canary-split` in `collectiveai-team/coro` (GitHub org; the HF org for this repo is collectiveai, not collectiveai-team) (onnx-canary-split backend, quantization= / decoder_quantization= selectors). The export/quantization recipes that produced these exact artifacts are also in that repo: coro/recipes/canary_split_decoder/, coro/recipes/canary_encoder_static_qdq/, coro/recipes/canary_decoder_dynamic_quantization/.

Status: comparative reference, not a recommended default

This is a research/benchmarking artifact from an internal ASR backend comparison, not a production recommendation. The project that produced it uses a different backend (onnx-asr's own default Parakeet integration) as its actual default — Canary is evaluated here specifically because it forces the transcription language natively (useful for language-constrained ASR use cases), which the default backend does not do reliably. If that's not a requirement for your use case, the base istupakov/canary-1b-v2-onnx (or an entirely different model) may be a better fit.

Files

License

CC-BY-4.0, same as nvidia/canary-1b-v2 and istupakov/canary-1b-v2-onnx. Attribution: NVIDIA (original model), istupakov (ONNX export this is built on), collective.ai (split decode-loop graph surgery + INT8 quantization in this repository).