CoolFace
Modelpublic

stephvax/sherpa-onnx-stt-int4

sourceHugging Facecc-by-4.0updated 2mo agoView on Hugging Face
1likes
Model Card

int4 STT weights for sherpa-onnx — Parakeet + Whisper small

Two things, for two different reasons — see each section.

1. parakeet/encoder.int4.onnx — a mirror

A mirror of the int4 encoder from `efederici/parakeet-tdt-0.6b-v3-onnx-int4`, itself quantized from `nvidia/parakeet-tdt-0.6b-v3` (CC-BY-4.0), with the NeMo `metadata_props` sherpa-onnx requires appended so it loads without any extra step.

Why this mirror exists

Two reasons, and only the second is about us.

It makes the file usable as-is. The upstream int4 encoder ships with no metadata_props, and sherpa-onnx refuses to load it:

offline-transducer-nemo-model.cc:InitEncoder:205
  'vocab_size' does not exist in the metadata

Eleven key/value pairs — copied verbatim from k2-fsa's export of the same NVIDIA checkpoint — are appended here. Note that url is load-bearing, not documentation: sherpa-onnx decides the model is a TDT (i.e. that the joiner emits duration logits on top of the vocabulary) by looking for tdt in that string. Drop it and the load fails with vocab_size: 8193 != output_size: 8198.

It removes a single point of failure. The upstream repo is a personal one. A personal repo can be deleted, renamed, or have its files replaced; an application that downloads from it at install time inherits that risk.

What was and was not changed

  • —The 391 MB of weights are byte-identical to upstream. The metadata is appended at the end of the file: metadata_props is field 14 of ModelProto and it is repeated, so in protobuf the appended entries concatenate onto the list. Nothing is re-encoded.
  • —Added: 315 bytes.
  • —sha256 of upstream encoder-model.int4.onnx: df4f1e5ff7a3af4e9d4b7078055164b11005e5d8a4c100e67f583c23975f7a31

Usage with sherpa-onnx

Pair it with the decoder, joiner and tokens from `csukuangfj/sherpa-onnx-nemo-parakeet-tdt-0.6b-v3-int8` — the vocabulary and decoder are byte-identical across exports of this checkpoint, so only the encoder differs.

python
rec = sherpa_onnx.OfflineRecognizer.from_transducer(
    encoder="encoder.int4.onnx",
    decoder="decoder.int8.onnx",
    joiner="joiner.int8.onnx",
    tokens="tokens.txt",
    model_type="nemo_transducer",
)

Why int4 rather than the usual int8

It is not about the bit count. The k2-fsa int8 encoder is quantized dynamically, per tensor, activations included (DynamicQuantizeLinear); this one is quantized weight-only, in blocks (MatMulNBits, activations untouched). Per-tensor dynamic quantization is the coarser approximation, and Conformer encoders are sensitive to it because their activations have very uneven dynamic range from layer to layer.

Measured on 200 French utterances of FLEURS (fr_fr), through sherpa-onnx, with the decoder/joiner/tokens held constant so the encoder is the only variable:

encoderWERsize
FP32 (reference)5.55 %2.4 GB
int8, per-tensor dynamic7.74 % (+2.18)652 MB
int4, block-wise weight-only6.60 % (+1.04)391 MB

Smaller and twice as close to the reference. Caveat worth stating: FLEURS is clean read speech, so this measures what quantization costs the model, not what a microphone costs it.

Requires ONNX Runtime ≥ 1.16 for MatMulNBits.

2. whisper-small/ — quantized here

Nobody publishes an int4 Whisper. This one was quantized from `csukuangfj/sherpa-onnx-whisper-small`'s FP32 export with onnxruntime's MatMulNBitsQuantizer (4 bits, block 64, accuracy_level=4), because on that size int4 measured better than the int8 normally shipped.

Same 200 French FLEURS utterances, sherpa-onnx, language pinned to fr:

WhisperFP32int8int4int4 − int8
tiny43.15 % / 152 MB55.51 % / 103 MB62.07 % / 95 MB+6.56
base27.17 % / 292 MB31.27 % / 160 MB31.84 % / 140 MB+0.57
small14.28 % / 969 MB15.29 % / 375 MB15.14 % / 287 MB−0.15

The pattern is monotone, and it has a cause: quantization damage shrinks as the model grows. Weight-only int4 only reaches MatMul; the embedding table it cannot touch is a large share of tiny and a small share of small. So int4 is a disaster on the smallest model and free on the largest — 0.15 point better and 87 MB lighter.

Only small is published here for that reason. tiny and base are better served by their existing int8 files.

Pair with small-tokens.txt from the k2-fsa repo — same vocabulary, nothing to quantize in it.

python
rec = sherpa_onnx.OfflineRecognizer.from_whisper(
    encoder="whisper-small/encoder.int4.onnx",
    decoder="whisper-small/decoder.int4.onnx",
    tokens="small-tokens.txt", language="fr", task="transcribe",
)

Whisper weights © OpenAI, MIT. Requires ONNX Runtime ≥ 1.16.