felixem/moonshine-streaming-tiny-optimized

1---2license: mit3base_model: UsefulSensors/moonshine-streaming-tiny4language:5- en6tags:7  - onnx8  - int89  - fp1610  - quantized11  - optimized12  - speech-recognition13  - asr14  - streaming15  - moonshine16library_name: onnxruntime17pipeline_tag: automatic-speech-recognition18---19 20# Moonshine Streaming Tiny — Optimized21 22Optimized variants of [UsefulSensors/moonshine-streaming-tiny](https://huggingface.co/UsefulSensors/moonshine-streaming-tiny), a 34M parameter streaming ASR model designed for real-time, on-device English speech recognition.23 24Based on: [Moonshine v2: Ergodic Streaming Encoder ASR](https://arxiv.org/abs/2602.12241)25 26## Optimized Variants27 28| Variant | Total Size | Size Reduction | Best For |29|---------|-----------|---------------|----------|30| **Original FP32** | 168.1 MB | — | Reference |31| **ONNX INT8** | 79.8 MB | **52%** | CPU deployment, edge devices |32| **FP16 SafeTensors** | 88.1 MB | **48%** | GPU inference |33| **ONNX FP32** | 297 MB | — | ONNX Runtime without quantization |34 35## Benchmark Results36 37Tested with 5 seconds of audio, generating up to 64 tokens:38 39| Variant | Avg Latency | RTF | Speedup vs FP32 CPU |40|---------|------------|-----|---------------------|41| **PyTorch FP16 (GPU)** | 47.7 ms | 0.0095 | **1.71x** |42| PyTorch INT8 (CPU) | 78.6 ms | 0.0157 | 1.03x |43| PyTorch FP32 (CPU) | 81.3 ms | 0.0163 | 1.00x (baseline) |44| ONNX FP32 (CPU) | 115.5 ms | 0.0231 | 0.70x |45| ONNX INT8 (CPU) | 153.2 ms | 0.0306 | 0.53x |46 47> **Note**: ONNX benchmarks include session overhead and were run on a single test. For production deployment with session reuse on real audio, ONNX Runtime typically provides better throughput, especially for long-running services. The Moonshine team reports 50ms response latency on Apple M3 with their C++ ONNX Runtime backend.48 49## File Structure50 51```52├── onnx_int8/                          # ONNX INT8 quantized (recommended for CPU)53│   ├── encoder_model_int8.onnx         # 9.8 MB54│   ├── decoder_model_int8.onnx         # 36 MB55│   ├── decoder_with_past_model_int8.onnx # 32 MB56│   ├── tokenizer.json57│   ├── config.json58│   └── quantize_config.json59├── onnx/                               # ONNX FP3260│   ├── encoder_model.onnx + .data61│   ├── decoder_model.onnx + .data62│   ├── decoder_with_past_model.onnx + .data63│   └── ...64└── fp16/                               # FP16 SafeTensors (for GPU)65    ├── model.safetensors               # 88.1 MB66    ├── config.json67    └── tokenizer.json68```69 70## Usage71 72### ONNX INT8 Inference (CPU — Recommended for Edge)73 74```bash75pip install onnxruntime numpy tokenizers76```77 78```python79import numpy as np80import onnxruntime as ort81from tokenizers import Tokenizer82 83MODEL_DIR = "onnx_int8"  # or download from this repo84BOS, EOS = 1, 285 86# Load models87opts = ort.SessionOptions()88opts.intra_op_num_threads = 489providers = ["CPUExecutionProvider"]90 91encoder = ort.InferenceSession(f"{MODEL_DIR}/encoder_model_int8.onnx", opts, providers=providers)92decoder = ort.InferenceSession(f"{MODEL_DIR}/decoder_model_int8.onnx", opts, providers=providers)93decoder_past = ort.InferenceSession(f"{MODEL_DIR}/decoder_with_past_model_int8.onnx", opts, providers=providers)94tokenizer = Tokenizer.from_file(f"{MODEL_DIR}/tokenizer.json")95 96# Prepare audio (16kHz float32, padded to multiple of 80 samples)97audio = np.random.randn(16000 * 5).astype(np.float32)  # replace with real audio98remainder = len(audio) % 8099if remainder:100    audio = np.pad(audio, (0, 80 - remainder))101 102audio_input = audio[np.newaxis, :]103attention_mask = np.ones_like(audio_input, dtype=np.int64)104 105# Encode audio106(enc_out,) = encoder.run(None, {107    "input_values": audio_input,108    "attention_mask": attention_mask,109})110 111# First decode step112outs = decoder.run(None, {113    "decoder_input_ids": np.array([[BOS]], dtype=np.int64),114    "encoder_hidden_states": enc_out,115})116logits, past_kvs = outs[0], outs[1:]117token = int(np.argmax(logits[0, -1, :]))118 119# Build KV cache mapping120dec_out_names = [o.name for o in decoder.get_outputs()][1:]121past_in_names = {i.name for i in decoder_past.get_inputs()} - {"decoder_input_ids", "encoder_hidden_states"}122 123kv_dict = {}124for name, tensor in zip(dec_out_names, past_kvs):125    mapped = name.replace("present_", "past_", 1)126    if mapped in past_in_names:127        kv_dict[mapped] = tensor128 129# Autoregressive decode loop130past_out_names = [o.name for o in decoder_past.get_outputs()][1:]131tokens = [token]132 133for _ in range(255):134    if token == EOS:135        break136    inputs = {137        "decoder_input_ids": np.array([[token]], dtype=np.int64),138        "encoder_hidden_states": enc_out,139    }140    inputs.update(kv_dict)141    outs = decoder_past.run(None, inputs)142    token = int(np.argmax(outs[0][0, -1, :]))143    tokens.append(token)144    145    kv_dict = {}146    for name, tensor in zip(past_out_names, outs[1:]):147        mapped = name.replace("present_", "past_", 1)148        if mapped in past_in_names:149            kv_dict[mapped] = tensor150 151text = tokenizer.decode(tokens)152print(text)153```154 155### FP16 PyTorch Inference (GPU)156 157```python158from transformers import MoonshineStreamingForConditionalGeneration, AutoProcessor159import torch160 161model = MoonshineStreamingForConditionalGeneration.from_pretrained(162    "felixem/moonshine-streaming-tiny-optimized",163    subfolder="fp16",164    torch_dtype=torch.float16,165).to("cuda")166 167processor = AutoProcessor.from_pretrained(168    "felixem/moonshine-streaming-tiny-optimized",169    subfolder="fp16",170)171 172# Process audio173inputs = processor(audio_array, return_tensors="pt", sampling_rate=16000)174inputs = {k: v.to("cuda", torch.float16) for k, v in inputs.items()}175 176generated_ids = model.generate(**inputs, max_new_tokens=128)177text = processor.decode(generated_ids[0], skip_special_tokens=True)178```179 180### PyTorch Dynamic INT8 (CPU — Quick Setup)181 182```python183import torch184from transformers import MoonshineStreamingForConditionalGeneration, AutoProcessor185 186model = MoonshineStreamingForConditionalGeneration.from_pretrained(187    "UsefulSensors/moonshine-streaming-tiny"188).eval()189 190# Quantize Linear layers to INT8191model = torch.quantization.quantize_dynamic(192    model, {torch.nn.Linear}, dtype=torch.qint8193)194 195processor = AutoProcessor.from_pretrained("UsefulSensors/moonshine-streaming-tiny")196inputs = processor(audio_array, return_tensors="pt", sampling_rate=16000)197generated_ids = model.generate(**inputs, max_new_tokens=128)198text = processor.decode(generated_ids[0], skip_special_tokens=True)199```200 201## ONNX Export Details202 203- **Encoder**: Exported with `torch.onnx.export(dynamo=True)` to handle vmap-based sliding-window attention masking204- **Decoder**: Separate models for first step (no KV cache) and autoregressive steps (with KV cache)205- **Quantization**: `onnxruntime.quantization.quantize_dynamic` with symmetric INT8, per-channel, reduce_range=True206 207### KV Cache Structure208 209Each decoder layer produces 4 KV tensors:210- `present_{layer}_self_key` / `present_{layer}_self_value`: Self-attention cache [B, 8, S, 40]211- `present_{layer}_cross_key` / `present_{layer}_cross_value`: Cross-attention cache [B, 8, T, 40]212 213For `decoder_with_past_model`, feed these back as `past_{layer}_*` inputs.214 215## Quantization Impact216 217Based on the [Edge-ASR paper](https://arxiv.org/abs/2507.07877) (Table 14), INT8 quantization on Moonshine Tiny has negligible WER impact:218 219| Config | Avg WER | vs FP32 |220|--------|---------|---------|221| FP32 baseline | 12.72% | — |222| **W8-A8 (INT8)** | **12.81%** | **+0.09%** |223| W4-A16 (SpQR) | 13.61% | +0.89% |224 225INT8 is the sweet spot for Moonshine Tiny — virtually no accuracy loss with ~50% model size reduction.226 227## Limitations228 229- English only230- Optimized for short utterances (streaming chunks of 1-5 seconds)231- ONNX models use external data files (`.onnx.data`) for FP32 variant232- The decoder uses autoregressive generation, so output latency scales with transcript length233 234## Citation235 236```bibtex237@article{kudlur2025moonshine,238  title={Moonshine v2: Ergodic Streaming Encoder ASR},239  author={Kudlur, Manjunath and King, Evan and Wang, James and Warden, Pete},240  journal={arXiv preprint arXiv:2602.12241},241  year={2025}242}243```244 245## License246 247MIT (same as base model)248