vbhar/whisperkit-hindi2hinglish-prime-coreml

whisperkit-hindi2hinglish-prime-coreml

A float16 Core ML conversion of `Oriserve/Whisper-Hindi2Hinglish-Prime` packaged for **WhisperKit** on Apple Silicon (macOS 14+ / iOS 17+).

It ships compiled .mlmodelc bundles, ready for on-device inference through WhisperKit's Swift API. For Python inference, use the upstream repo linked above.

Output script: romanized Hinglish (Latin script). Hindi speech comes back transliterated into English letters, not Devanagari.

Usage

import WhisperKit

let pipe = try await WhisperKit(
    WhisperKitConfig(
        model: "Oriserve_Whisper-Hindi2Hinglish-Prime_fp16",
        modelRepo: "vbhar/whisperkit-hindi2hinglish-prime-coreml"
    )
)
let result = try await pipe.transcribe(audioPath: "meeting.wav")

Or download the variant folder directly and point WhisperKit at a local path:

hf download vbhar/whisperkit-hindi2hinglish-prime-coreml \
  --include "Oriserve_Whisper-Hindi2Hinglish-Prime_fp16/*" \
  --local-dir ./models

Layout

Oriserve_Whisper-Hindi2Hinglish-Prime_fp16/
├── MelSpectrogram.mlmodelc/
├── AudioEncoder.mlmodelc/     # ~1.2 GB weights
├── TextDecoder.mlmodelc/      # ~1.7 GB weights
├── config.json
└── generation_config.json

No quantized variants and no decoder prefill data are included — float16 only, by choice (see A note on quantization below).

Attribution and modifications

Lineage:

All credit for the model itself belongs to Oriserve. Licensed under the Apache License, Version 2.0; see <https://www.apache.org/licenses/LICENSE-2.0>.

Statement of modifications (Apache-2.0 §4(b)): the weights were converted from float32 to float16 and compiled to Core ML for the Apple Neural Engine. They are otherwise unmodified — no fine-tuning, no distillation, no pruning, no vocabulary change, no re-training. The only non-weight change made during conversion was supplying alignment_heads metadata that the source repo ships as null (see Conversion gotcha below); those heads are the standard whisper-large-v3 values and affect word-timestamp attention selection only, not transcription weights.

Reproducing this conversion

Converted with `argmaxinc/whisperkittools` under Python 3.11 (3.14 has no working coremltools/torch wheels as of this writing):

python3.11 -m venv .venv && source .venv/bin/activate
pip install git+https://github.com/argmaxinc/whisperkittools.git

whisperkit-generate-model \
  --model-version Oriserve/Whisper-Hindi2Hinglish-Prime \
  --output-dir out \
  --repo-path-suffix fp16

float16, no quantized variants, no decoder prefill data.

Conversion gotcha: alignment_heads is null

Oriserve/Whisper-Hindi2Hinglish-Prime ships generation_config.json with:

"alignment_heads": null

The whisperkittools decoder trace uses alignment_heads to build the alignment_heads_weights output. With null, the traced decoder emits 3 outputs while the conversion declares 4, and the run dies with:

ValueError: Number of outputs provided, 4, do not match the number of outputs
produced by the model, 3

Fix: fall back to the standard whisper-large-v3 alignment heads — 10 [layer, head] pairs — before tracing:

generation_config.alignment_heads = [
    [7, 0], [10, 17], [12, 18], [13, 12], [16, 1],
    [17, 14], [19, 11], [21, 4], [24, 1], [25, 6],
]

This is the correct fallback, not a guess: the model is an unmodified-architecture whisper-large-v3 fine-tune, and these pairs are byte-identical to the ones the sibling `vbhar/whisperkit-hinglish-large-v3-coreml` build carries from its own source. The values are baked into the generation_config.json in this repo, so a consumer of this repo never sees the failure — only someone re-converting from the Oriserve source does.

A note on quantization

Float16 was chosen deliberately. During this work, a quantized Core ML build of these weights was observed to emit the literal string nan on real code-switched Hindi/English speech, while passing cleanly on short English clips such as WhisperKit's bundled jfk.wav.

The asymmetry is the point worth noting: quantization error is input-dependent, so a short smoke test — the normal way to sanity-check a conversion — can pass on a build that fails on the audio you actually care about. If you evaluate a quantized Whisper conversion, test it on your own code-switched audio at realistic length, not on a short clip.

No quantized variant is published here, because none was validated on that kind of input.

Limitations

• —~2.9 GB on disk. The variant folder is the full large-v3 parameter count at float16.
• —First load is slow. The Neural Engine compiles the model once on first use: roughly 115–190 s on an M-series Mac. Subsequent warm loads are ~1–3 s. Cache the compiled artifacts; do not put the first load on a user-visible path without a progress indicator.
• —Apple Silicon only. Requires macOS 14+ / iOS 17+ and an ANE. No Intel Mac support, no CUDA, no CPU-only fallback path worth using.
• —Script behaviour is not configurable. This model romanizes Hindi into Latin script. If you need Devanagari for the Hindi portions, this is the wrong model — use `vbhar/whisperkit-hinglish-large-v3-coreml` instead. Picking the wrong one of the two is the most likely mistake with this pair.
• —Inherits every bias and domain limitation of the upstream fine-tune, which was trained largely on Indian call-centre-style speech.

Verified usage

Tested live end-to-end in a real macOS menu-bar meeting-notes app (mic + system audio, streaming transcription) on natural Hindi/English code-switched speech.

Hindi is returned transliterated into Latin script and English is returned as-is, mixed inline within the same sentence.