parismitaglobalsolutions/indicconformer-sherpa-onnx

Indian + world language speech recognition: Android-ready sherpa-onnx ONNX exports

Quantized ONNX speech recognition models for on-device use with sherpa-onnx, small enough for budget Android phones and needing no internet once downloaded.

• —All 22 official Indian languages from AI4Bharat's IndicConformer
• —English from sherpa-onnx's pretrained NeMo fast-conformer CTC model
• —Hinglish (Hindi-English code-switched) from Oriserve's fine-tuned Whisper models
• —13 world languages (Chinese, Dutch, French, German, Indonesian, Italian, Japanese, Korean, Polish, Portuguese, Russian, Spanish, Turkish) from two shared multilingual OpenAI Whisper exports, re-exported so they return real per-token timestamps

Used in production

These exact files power [AI Subtitle Generator](https://parismitaglobalsolutions.com/apps/ai_subtitle_generator/), an Android app that makes subtitles and captions on the phone in 36 languages plus Hinglish (Google Play). Every model is downloaded on demand at runtime rather than bundled, which keeps the app install small.

Purpose

The original checkpoints are excellent but built for server-side PyTorch inference, and aren't practical to ship inside a mobile app. This repo re-packages them as quantized ONNX files that:

• —Run entirely on-device, with no server, no API key, and no internet needed once downloaded.
• —Are small enough to download and run on budget Android hardware.
• —Work directly with sherpa-onnx's Kotlin/Java API on Android, or its Python API anywhere else.

This is a community effort. The goal is to let any Indian-language or multilingual app (subtitle generators, voice assistants, accessibility tools) add on-device speech recognition without repeating the export and quantization work.

What's in this repo

Indian languages (AI4Bharat IndicConformer)

All 22 languages of India's Eighth Schedule, exported, quantized, and validated.

The scripts come straight from each model's slice of the shared tokenizer vocabulary. Four are easy to get wrong from memory: Kashmiri outputs Perso-Arabic (right-to-left), Sindhi outputs Devanagari (left-to-right), Manipuri outputs Meetei Mayek and Santali outputs Ol Chiki. If you render these transcripts, handle Kashmiri and Urdu as RTL and make sure your platform has fonts for Meetei Mayek and Ol Chiki.

English

en/ holds sherpa-onnx's own sherpa-onnx-nemo-fast-conformer-ctc-en-24500 model with int8 quantization applied, and its own tokens.txt.

Hinglish (Hindi-English code-switched)

Two models for natural code-switched speech: a speaker mixing Hindi and English mid-sentence, the way people actually talk in India. They are built on Oriserve's Whisper-Hindi2Hinglish fine-tunes and output Romanized Hinglish text directly (e.g. "Doston, nested aur multilevel if statement ke spoken tutorial mein aapka svaagat hai.") rather than Devanagari.

Apex is meaningfully more accurate, especially on noisy, conversational audio (Indic-Voices), at the cost of a much larger download. Swift is far smaller and faster. Both were also validated by ear against real Hinglish audio, not just benchmark numbers.

World languages (shared multilingual Whisper)

Two official OpenAI checkpoints, each exported once and shared by every world language. The language is chosen at decode time (language="es", "fr", …), so one download serves them all.

• —whisper-multilingual-apex/: openai/whisper-large-v3-turbo (encoder ~645 MB + decoder ~345 MB int8)
• —whisper-multilingual-swift/: openai/whisper-small (encoder ~105 MB + decoder ~250 MB int8)

Why re-export instead of using sherpa-onnx's published Whisper models: those exports have no cross_attention_weights output, so they return zero timestamps even with token timestamps enabled. That was confirmed by test. They transcribe well, but you can't build subtitle timing from them. These re-exports include that output and produce per-token timestamps.

Benchmark: FLEURS test split, 10 clips per language, exactly these two checkpoints, Whisper's BasicTextNormalizer. Scores are word error rate, except character error rate for Japanese, Korean and Chinese.

FLEURS is clean read speech, so real-world audio (music, noise, overlapping speakers) will score worse. Ten clips per language is a sanity check, not a full evaluation. For Chinese we recommend Apex only: Swift made real word errors. Whisper can also drift between simplified and traditional characters.

A note on fine-tunes: for Turkish, plain whisper-large-v3-turbo (8.8% WER on an earlier 8-clip FLEURS run) beat a popular Turkish fine-tune (12.4%). Fine-tunes trained on Common Voice tend to score well there and worse on unseen audio, so test on FLEURS too before switching.

Repo layout

tokens.txt <- shared by all 22 AI4Bharat Indian languages available_languages.json <- the language manifest used by the AI Subtitle Generator app

as/ bn/ brx/ doi/ gu/ hi/ kn/ kok/ ks/ mai/ ml/ mni/ mr/ ne/ or/ pa/ sa/ sat/ sd/ ta/ te/ ur/ model.int8.onnx <- one per Indian language

en/model.int8.onnx en/tokens.txt <- English has its own, unrelated vocabulary

hi-hinglish-apex/encoder.int8.onnx hi-hinglish-apex/decoder.int8.onnx hi-hinglish-apex/tokens.txt hi-hinglish-swift/ (same three files)

whisper-multilingual-apex/encoder.int8.onnx whisper-multilingual-apex/decoder.int8.onnx whisper-multilingual-apex/tokens.txt whisper-multilingual-swift/ (same three files)

• —Indian languages: the root tokens.txt is identical for all 22 (verified, not assumed), so download it once.
• —English: a different model family with its own tokens.txt inside en/.
• —Whisper models (Hinglish and multilingual): separate encoder and decoder graphs, so download all three files per variant. The two multilingual variants share the same vocabulary.
• —Leftover files: all int8 files are self-contained. Any encoder.weights / decoder.weights files in the Hinglish folders are leftovers from an earlier export and are not needed.

How to use it

Python (sherpa-onnx)

import sherpa_onnx

# Any Indian language (shared tokens.txt) recognizer = sherpaonnx.OfflineRecognizer.fromnemoctc( model="hi/model.int8.onnx", # swap for any language folder above tokens="tokens.txt", numthreads=2, decodingmethod="greedysearch", )

# English (its own tokens.txt) recognizeren = sherpaonnx.OfflineRecognizer.fromnemoctc( model="en/model.int8.onnx", tokens="en/tokens.txt", numthreads=2, decodingmethod="greedy_search", )

# Hinglish (Whisper-based) recognizerhinglish = sherpaonnx.OfflineRecognizer.fromwhisper( encoder="hi-hinglish-apex/encoder.int8.onnx", # or hi-hinglish-swift/... decoder="hi-hinglish-apex/decoder.int8.onnx", tokens="hi-hinglish-apex/tokens.txt", numthreads=2, decodingmethod="greedysearch", language="hi", task="transcribe", )

# World languages (shared multilingual Whisper: pick the language at decode time) recognizeres = sherpaonnx.OfflineRecognizer.fromwhisper( encoder="whisper-multilingual-apex/encoder.int8.onnx", # or whisper-multilingual-swift/... decoder="whisper-multilingual-apex/decoder.int8.onnx", tokens="whisper-multilingual-apex/tokens.txt", numthreads=2, decodingmethod="greedysearch", language="es", # "fr", "de", "ja", "tr", ... task="transcribe", )

stream = recognizer.createstream() stream.acceptwaveform(16000, audio) # audio: float32 numpy array, 16 kHz mono recognizer.decode_stream(stream) print(stream.result.text)

Per-token timestamps for the Whisper models come back in stream.result.timestamps when you use a sherpa-onnx build with Whisper token-timestamp support (PR #2945).

Android (Kotlin)

Use sherpa-onnx's Android AAR:

• —Indian languages and English: OfflineRecognizer with OfflineNemoEncDecCtcModelConfig.
• —Hinglish and world languages: OfflineRecognizer with OfflineWhisperModelConfig.

The Kotlin API mirrors the Python one closely. Download the model and matching tokens file to app-writable storage at runtime, then point the config at those file paths. No bundled assets or special permissions are needed. This is exactly how AI Subtitle Generator uses this repo.

Downloading files directly

Every file is a plain public HTTPS download, with no auth required:

https://huggingface.co/parismitaglobalsolutions/indicconformer-sherpa-onnx/resolve/main/tokens.txt https://huggingface.co/parismitaglobalsolutions/indicconformer-sherpa-onnx/resolve/main/hi/model.int8.onnx https://huggingface.co/parismitaglobalsolutions/indicconformer-sherpa-onnx/resolve/main/en/model.int8.onnx https://huggingface.co/parismitaglobalsolutions/indicconformer-sherpa-onnx/resolve/main/en/tokens.txt https://huggingface.co/parismitaglobalsolutions/indicconformer-sherpa-onnx/resolve/main/hi-hinglish-apex/encoder.int8.onnx https://huggingface.co/parismitaglobalsolutions/indicconformer-sherpa-onnx/resolve/main/hi-hinglish-apex/decoder.int8.onnx https://huggingface.co/parismitaglobalsolutions/indicconformer-sherpa-onnx/resolve/main/whisper-multilingual-apex/encoder.int8.onnx https://huggingface.co/parismitaglobalsolutions/indicconformer-sherpa-onnx/resolve/main/whisper-multilingual-apex/decoder.int8.onnx https://huggingface.co/parismitaglobalsolutions/indicconformer-sherpa-onnx/resolve/main/whisper-multilingual-apex/tokens.txt

Swap the folder name for any other language or variant.

How these were made

The 22 Indian languages:

• —Source: AI4Bharat's indicconformer_stt_<lang>_hybrid_ctc_rnnt_large checkpoints (Conformer-Large, hybrid CTC+RNNT, 120M params, with an aggregate tokenizer shared across all 22 languages).
• —Export: ONNX, using the CTC head only (much cheaper on mobile than RNNT). Each language's vocabulary mask is baked into the graph, so any consumer can do plain greedy argmax decoding safely.
• —Quantization: onnxruntime dynamic quantization, restricted to MatMul ops (int8). Full-model int8 and fp16 both hurt accuracy. MatMul-only int8 cuts size by roughly 60% with no measurable quality loss.
• —Validation: exported ONNX logits compared against PyTorch's own on identical input; the language mask confirmed exactly -inf outside its slice in both fp32 and int8; int8 vs fp32 drift checked. The earlier languages were also checked end-to-end against reference transcripts on desktop and real Android hardware.
• —Odia: AI4Bharat's published Odia checkpoint ships one corrupt tokenizer file (precompiled_charsmap is invalid), reproducible on a fresh download. Because all 22 checkpoints carry the same aggregate tokenizer set, it was repaired by replacing the one bad file with its byte-identical counterpart from another language's checkpoint, keeping Odia's filename. The resulting vocabulary slice was confirmed to be real Odia subwords.

English:

• —Source: sherpa-onnx's pretrained sherpa-onnx-nemo-fast-conformer-ctc-en-24500, already a valid ONNX export.
• —Quantization: the same MatMul-only int8 recipe, validated against sherpa-onnx's bundled test audio.

Hinglish (Apex / Swift):

• —Source: Oriserve's Whisper-Hindi2Hinglish-Apex (Whisper Large-V3 Turbo, 807M params, ~700 hours of Indian-accented Hindi/Hinglish) and Whisper-Hindi2Hinglish-Swift (Whisper-base, 72.6M params, ~550 hours), both Apache 2.0.
• —Conversion: from Hugging Face transformers format to an OpenAI-format checkpoint using each repo's own convert_hf2openai.json map, then verified to load via whisper.load_model().
• —Export: sherpa-onnx's scripts/whisper/export-onnx-with-attention.py (PR #2945), with the DTW alignment heads for each fine-tune discovered empirically (they aren't in the script's built-in table).
• —Quantization and validation: MatMul-only int8, then validated by ear against real Hinglish audio, with per-token timestamps checked for correct pacing.

World languages (whisper-multilingual Apex / Swift):

• —Source: official openai/whisper-large-v3-turbo (MIT) and openai/whisper-small (Apache 2.0).
• —Export: the same export-onnx-with-attention.py. Both checkpoints are already in its alignment-heads table, so no head discovery was needed. The script was patched to force the legacy (non-dynamo) ONNX exporter and to quantize in a separate step, avoiding out-of-memory on the large model.
• —Timestamp check: every export must pass before upload. Tokens must come back with timestamps, with no repeats and sensible coverage, and the two models' timings must agree within tens of milliseconds.
• —Known quirk: the first token can pin to 0.00 s, with the second at the real speech start, so a clip with a long silent intro may show its first cue early.

Testing / sample audio

For real Indian-language clips, the Indian Languages Audio Dataset on Kaggle is a useful public source. For world languages, Google's FLEURS dataset is the one used for the benchmark above.

Reproduce this yourself / convert more languages

The notebooks used to produce these models are at the repo root.

AI4Bharat Indian languages: `ai4bharat_export_pipeline.ipynb`.

1. 1.Open it in Google Colab and set the runtime to CPU.
2. 2.Change LANG_CODE near the top.
3. 3.Run the cells top to bottom.

Two Colab traps, both handled in the notebook:

• —Install order: NeMo's install leaves an incompatible Levenshtein build, so it is force-reinstalled after NeMo.
• —Restart: a Runtime → Restart session is required between install and first import.

Hinglish (Apex / Swift): three notebooks per variant, run in this order:

1. 1.`hinglish_apex_export_pipeline.ipynb` / `hinglish_swift_export_pipeline.ipynb`: converts the checkpoint to OpenAI format and does a first export. Kept for reference; superseded by step 3 for real timestamps.
2. 2.`hinglish_apex_alignment_heads.ipynb` / `hinglish_swift_alignment_heads.ipynb`: discovers the DTW alignment heads for these fine-tuned checkpoints.
3. 3.`hinglish_apex_attention_export.ipynb` / `hinglish_swift_attention_export.ipynb`: produced the published hi-hinglish-apex/ and hi-hinglish-swift/ files, with real per-word timestamps.

Any other Whisper checkpoint follows the same pattern:

1. 1.Convert it to OpenAI format if needed.
2. 2.Use the built-in alignment heads, or discover them for a fine-tune.
3. 3.Export with attention.
4. 4.Confirm timestamps actually come back before using the model.

Getting a Hugging Face token to run the notebook

AI4Bharat's source model repos are gated: free to use, but you must accept their terms once per language before downloading. To run ai4bharat_export_pipeline.ipynb yourself:

1. 1.Create a free account at huggingface.co if you don't have one.
2. 2.Visit https://huggingface.co/ai4bharat/indicconformer_stt_<lang>_hybrid_ctc_rnnt_large for the language you want, log in, and click "Agree and access repository".
3. 3.Create a Read token at huggingface.co/settings/tokens.
4. 4.In Google Colab: click the key icon (🔑) in the left sidebar → Add new secret → name it exactly HF_TOKEN → paste your token → enable Notebook access.
5. 5.Never paste the token value into a code cell. The notebook reads it via userdata.get('HF_TOKEN').

Credit and license

• —The 22 Indian language models are built on the work of AI4Bharat (IIT Madras). All model weights, architecture and training are theirs; this repo only re-packages their published checkpoints for on-device mobile use.
• —The English model is built on NVIDIA NeMo's fast-conformer CTC checkpoint, converted to ONNX by the sherpa-onnx / k2-fsa project.
• —The Hinglish models are built on Oriserve's Whisper-Hindi2Hinglish-Apex and Whisper-Hindi2Hinglish-Swift fine-tunes, themselves built on OpenAI Whisper.
• —The world-language models are OpenAI's official whisper-large-v3-turbo (MIT) and whisper-small (Apache 2.0, per its Hugging Face model card), with ONNX export and int8 quantization only.
• —Timestamp-capable Whisper export uses sherpa-onnx's export-onnx-with-attention.py.

Full credit for the speech recognition research and training belongs to AI4Bharat, NVIDIA NeMo, OpenAI, Oriserve and the sherpa-onnx/k2-fsa team. Each export keeps its source model's licence, as listed in the table at the top.

Contributing

If you convert additional languages with these pipelines and want to add them here, open a discussion on this repo.