bitsydarel/moonshine-tiny-onnx-mobile

Moonshine Tiny ONNX (Mobile)

Mobile-compatible Moonshine Tiny for on-device speech recognition on iOS and Android.

This model re-exports the encoder from onnx-community/moonshine-tiny-ONNX using dynamic quantization to eliminate ConvInteger operators that are blocked on ONNX Runtime Mobile. All other files (decoders, tokenizer, config) are unchanged from the upstream repository.

Why This Model Exists

The upstream onnx-community/moonshine-tiny-ONNX int8 encoder uses Optimum's default static quantization, which produces ConvInteger operators (ONNX opset 10). These operators are not registered in ORT Mobile on iOS or Android, causing a crash at session creation:

onnxruntime::Model::Model: ... ConvInteger is not a registered function/op

This repository replaces the encoder with a dynamically quantized version that uses MatMulInteger + DynamicQuantizeLinear — both fully supported on ORT Mobile — while preserving near-lossless accuracy.

Accuracy

Encoder output comparison on 200 LibriSpeech dev-clean-2 utterances:

Dynamic quantization computes activation ranges on-the-fly via DynamicQuantizeLinear, making it robust to variable-length speech inputs. Static quantization fixes activation ranges at export time using calibration data, which generalizes poorly to utterances outside the calibration distribution.

Supported Platforms

Usage

import onnxruntime as ort
from huggingface_hub import hf_hub_download

# Download the int8 encoder
encoder_path = hf_hub_download(
    "bitsydarel/moonshine-tiny-onnx-mobile",
    "onnx/encoder_model_int8.onnx"
)

# Create session
session = ort.InferenceSession(
    encoder_path,
    providers=["CPUExecutionProvider"]
)

# Verify inputs
for inp in session.get_inputs():
    print(f"{inp.name}: {inp.shape} ({inp.type})")

Model Files

This repository contains two complete model configurations — FP32 and INT8 — plus shared tokenizer and config files. All decoders are merged variants with KV-cache support via the use_cache_branch input. Basic (non-cached) decoders are not included.

Model Configurations

ONNX Files

Shared Files

KV-Cache Decoder Usage

Both merged decoders use the use_cache_branch boolean input to control KV-cache behavior:

• —Step 0 (use_cache_branch=false): Computes fresh key-value pairs from encoder hidden states. All past_key_values inputs must still be provided (use zero sequence length).
• —Step 1+ (use_cache_branch=true): Reuses cached key-value pairs for faster autoregressive decoding. Decoder KV-cache grows each step; encoder KV-cache is frozen after step 0.

Important: The merged decoder outputs invalid encoder cache when use_cache_branch=true. Always preserve the encoder cache from step 0 and ignore encoder cache outputs on subsequent steps.

Technical Details

Why Dynamic over Static Quantization?

Speech-to-text models process variable-length audio inputs. Static quantization (QDQ format) fixes activation ranges at export time using calibration data, which makes assumptions about the distribution of input values. When inference-time inputs differ from calibration data — common with speech of varying length, volume, and content — activation clipping causes significant accuracy loss.

Dynamic quantization computes activation ranges on-the-fly for each inference call using DynamicQuantizeLinear. This adds minimal latency overhead (< 1ms per encoder pass on Apple A17) while adapting to every input. For the Moonshine encoder, this preserves 0.9986 cosine similarity vs FP32, compared to 0.634 with static quantization.

Quantization Method

from onnxruntime.quantization import quantize_dynamic, QuantType

quantize_dynamic(
    model_input="encoder_model.onnx",
    model_output="encoder_model_int8.onnx",
    weight_type=QuantType.QInt8,
    op_types_to_quantize=["MatMul"],  # Only quantize MatMul, leave Conv in fp32
)

Only MatMul operations are quantized. Convolution layers remain in FP32 to avoid ConvInteger operators entirely.

Operator Compatibility

Limitations

• —English only — Moonshine Tiny is trained on English speech data
• —27M parameters — Smallest Moonshine variant; for higher accuracy, consider Moonshine Base (61M params)
• —Short-form audio — Optimized for utterances up to ~30 seconds; for longer audio, segment before transcribing
• —Encoder-only re-export — Only the encoder was re-exported with dynamic quantization; decoder files are unchanged from upstream

Citation

@article{jeffries2024moonshine,
  title={Moonshine: Speech Recognition for Live Transcription and Voice Commands},
  author={Jeffries, Nat and Birch, Evan and Zhang, Micha\l{}},
  journal={arXiv preprint arXiv:2410.15608},
  year={2024}
}

License

This model is released under the MIT License, consistent with the original Moonshine model license.

Acknowledgments

• —Useful Sensors for the Moonshine model family
• —Hugging Face Optimum for ONNX export tooling
• —ONNX Runtime for cross-platform inference