soniqo/Nemotron-3.5-ASR-Streaming-Multilingual-0.6B-ONNX-FP16

Nemotron-3.5-ASR-Streaming-Multilingual-0.6B — ONNX (FP16)

Cache-aware streaming multilingual speech recognition. A 0.6 B FastConformer-RNNT encoder with a 128-slot language prompt (one slot per language/locale), exported to ONNX in FP16. FP16 is lossless versus the FP32 source (identical WER/CER) at half the download size, and is hardware-accelerated on GPU / NPU / Android-NNAPI.

• —Architecture: cache-aware FastConformer encoder (24 layers, 1024 hidden, 8× subsampling) + RNN-T decoder/joint
• —Streaming: 320 ms chunk, 240 ms lookahead, left attention context 56, right context 3
• —Languages: 100+ via the prompt dictionary (languages.json); benchmarked on 6 below
• —Audio: 16 kHz mono, 128-bin log-mel front end

Model

Files

Performance

FLEURS test, 320 ms streaming, CPU, n=30 per language. FP16 is bit-equivalent to FP32. For Japanese, CER is the meaningful metric (no word boundaries).

Resource profile (8.4 s utterance, ONNX Runtime CPU): encoder ~87 ms/chunk (RTF ~0.27), peak RSS ~3.4 GB. Note: ONNX Runtime up-converts FP16→FP32 on CPU (no native FP16 CPU kernels), so FP16's runtime wins are realized on GPU / NPU / NNAPI — on CPU its benefit is the smaller download. For lowest CPU latency/RAM, use the INT8 build.

Usage

import onnxruntime as ort

so = ort.SessionOptions()
enc = ort.InferenceSession("encoder.onnx", so, providers=["CPUExecutionProvider"])
dec = ort.InferenceSession("decoder.onnx", so, providers=["CPUExecutionProvider"])
joint = ort.InferenceSession("joint.onnx", so, providers=["CPUExecutionProvider"])

# Pick the language prompt slot from languages.json, e.g. "en-US" -> 0, "ja-JP" -> 10.
# Front end: 128-bin log-mel (n_fft=512, win=400, hop=160, preemph=0.97), 16 kHz mono.
# Streaming contract (per chunk): feed 320 ms of audio + the carried encoder caches
# (attention / conv / pre-cache), then run the RNN-T greedy loop over the 4 emitted frames.
# Port wiring for the encoder caches is described by config.json's "streaming" block.

Production streaming, cache management and RNN-T greedy decoding are handled by the [speech-android](https://github.com/soniqo/speech-android) SDK.

Source

Converted from [nvidia/nemotron-3.5-asr-streaming-0.6b](https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b) (NVIDIA NeMo). Licensed under the NVIDIA Open Model License.

WebGPU / browser

The encoder emits its per-layer streaming-cache concatenation as a tree of ≤6-input `Concat` nodes (instead of one 24-input Concat), so no node exceeds WebGPU's maxStorageBuffersPerShaderStage limit (8). This lets onnxruntime-web run the encoder under the WebGPU execution provider — including Compatibility mode and mobile GPUs that cap at 8 storage buffers per shader stage. The rewrite is numerically identical to a standard graph (concatenation is associative) and has no effect on CPU/NNAPI/CUDA runtimes. (INT8 is not WebGPU-compatible — its ConvInteger op is unsupported there; use this FP16 build for the browser.)

Related models

Links

• —speech-android — Android SDK
• —speech-core — on-device inference core (C++)
• —soniqo.audio — website
• —blog — blog