mlboydaisuke/Whisper-base-ExecuTorch

1---2license: apache-2.03tags:4- executorch5- xnnpack6- pte7- on-device8- automatic-speech-recognition9base_model:10- openai/whisper-base11base_model_relation: quantized12---13# Whisper-base — ExecuTorch (encoder + decoder)14 15Speech recognition in two `.pte` files: the encoder runs once per 30-second window, the16decoder once per generated token. Putting them in one graph would re-encode the audio on17every step.18 19| graph | build | file | size (MB) | corr vs fp32 eager | ms | eager ms |20|---|---|---|---|---|---|---|21| encoder | XNNPACK fp32 | `whisper_base_encoder_xnnpack_fp32.pte` | 82.4 | 1.000000 | 120.1 | 43.8 |22| encoder | XNNPACK fp16 | `whisper_base_encoder_xnnpack_fp16.pte` | 43.1 | 0.999997 | 217.6 | 43.0 |23| encoder | XNNPACK int8 | `whisper_base_encoder_xnnpack_int8.pte` | 25.9 | 0.999374 | 114.3 | 45.7 |24| encoder | Core ML | `whisper_base_encoder_coreml_all.pte` | 41.4 | 0.999983 | 28.1 | 44.2 |25| decoder | XNNPACK fp32 | `whisper_base_decoder_xnnpack_fp32.pte` | 314.4 | 1.000000 | 36.1 | 20.0 |26| decoder | XNNPACK fp16 | `whisper_base_decoder_xnnpack_fp16.pte` | 157.4 | 0.999981 | 77.3 | 19.4 |27| decoder | Core ML | `whisper_base_decoder_coreml_all.pte` | 104.2 | 0.999861 | 5.0 | 20.0 |28 29Every file takes and returns fp32 tensors (token ids stay int64), so any encoder pairs with30any decoder. The lightest working pair is 130.1 MB.31 32- **Source**: [openai/whisper-base](https://huggingface.co/openai/whisper-base)33- **License**: Apache-2.034- **Encoder input**: log-mel spectrogram `[1, 80, 3000]` — 30 s at 16 kHz, 80 mel bins, hop35  160, window 400, exactly what `WhisperFeatureExtractor` produces36- **Decoder input**: the encoder output plus `decoder_input_ids [1, 128]` int64,37  left-aligned and padded. Start with `<|startoftranscript|>`, a language token,38  `<|transcribe|>`, `<|notimestamps|>`.39 40## Decoding41 42No KV cache: the decoder is a static graph over a fixed 128-token window, so a greedy step43is take `argmax` of row `len-1`, append it, run again. Stop at `<|endoftext|>` (50257). 12844tokens covers a 30-second window of ordinary speech; past that, start a new window.45 46That costs a full 128-position forward pass per token, which is the price of a static graph47that runs unchanged across runtimes and precisions.48 49## Verification (Mac arm64, executorch 1.4.0, torch 2.13.0)50 51The two wrappers compose back to `WhisperForConditionalGeneration` exactly — max_abs_diff52**0.000e+00** — and every graph matches torch fp32 eager at the correlations above. Timings53are medians over 5 runs in one process: a relative reference, not a device number.54 55## Two things worth knowing about the sizes56 57**The decoder `.pte` is larger than the decoder's weights.** Whisper ties `proj_out.weight`58to `decoder.embed_tokens.weight`, but the two uses need different representations: an59embedding table the portable kernels index into, and the same values packed into the XNNPACK60delegate's blob for the output matmul. Tying them in PyTorch does not tie them here.61Referencing the weight through `F.linear` instead of the `proj_out` module does not either —62exported both ways, whisper-tiny's decoder comes out at 198.0 MB exactly.63 64**The decoder's int8 build is not shipped.** Dynamic int8 quantizes the linear weights and65leaves the token embedding table in fp32, and that table is 106.2 MB — 51,865 tokens at66512 dimensions. On this size that table is most of the file, so int8 lands at 159.6 MB against fp16's 157.4 MB — larger, because fp16 halves the table too. It converts and holds, but nothing would pick it, so it is not shipped.67 68Until recently there was no decoder int8 build at all, and this card said PT2E was observing69the int64 `decoder_input_ids`. That was wrong on both halves.70`XNNPACKQuantizer.transform_for_annotation` rewrites every scalar argument of71`add.Tensor`/`mul.Tensor` as `torch.tensor(float(arg))` whatever the node's dtype — one line72in ExecuTorch's `backends/xnnpack/quantizer/xnnpack_quantizer_utils.py`, still present on73main. In this decoder the casualty is `position_ids = torch.arange(...) + past_key_values_length`74(`modeling_whisper.py:749`, `past_key_values_length` being a python `int`): it comes back75float32, and the failure lands on `self.weight[position_ids]` — the **position** embedding76lookup, not the token ids, and no observer involved. Measured by running `prepare_pt2e`77with an empty quantizer and printing the failing node.78 79### Checked in the task's own units80 81Correlation is a first filter. These are the numbers that decide:82 83- **encoder int8** — measured end to end — word error rate against the fp32 encoder: mean WER 0.0% (worst clip 0.0%) over 5 spoken sentences, int8 encoder against the fp32 encoder with the same fp32 decoder and the same waveform; the fp32 arm transcribes all five correctly, so the comparison is against a working control rather than against noise.84 85The sensitivity of that test, measured by injecting random noise into whisper-tiny's encoder output: rel_l2 0.03 (what int8 actually costs) and 0.10 both give WER 0.000; 0.20 and 0.40 give 0.025. Five clean sentences leave headroom, so a pass means *does not break the transcript*, not *indistinguishable at any error level*.86 87## Conversion88 89```bash90python convert/export_whisper.py base91```92 93The ExecuTorch tree ships a single-graph Whisper example under `examples/models/whisper`;94this is that model with the halves separated.95 96(conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models))97