Sanat-agrwl/indic-whisper-ctc-timestamps

Indic Whisper with a character CTC head: transcript and word timestamps in one pass

Whisper large-v2 fine-tuned for Hindi, with a character-level CTC head over the same encoder. Audio goes through the encoder once: the attention decoder reads the result to produce the transcript, and the CTC head reads the same tensor to place word boundaries by forced alignment at 20 ms resolution. No second model and no second encoder pass, unlike pipelines that run a separate wav2vec2 aligner.

This is configuration E of the accompanying paper.

Usage

python word_timestamps.py clip.wav --model Sanat-agrwl/indic-whisper-ctc-timestamps --head ctc_head_E.pt

Outputs the transcript followed by one line per word with start and end times in seconds.

Calibration

Boundaries from a CTC head are systematically tight: words start about 38 ms late and end about 100 ms early relative to human annotation. Subtracting those two constants roughly halves the error, from 74.5 to 43.0 ms mean, measured out-of-sample against human boundaries. Pass --calibrate to apply them. Because the offsets apply independently, calibrated spans of adjacent words can overlap slightly.

What the timestamp numbers mean

The 20.8 ms above is agreement with an external wav2vec2 aligner, not accuracy. Measured against human annotation, this head and that aligner are statistically indistinguishable (74.5 vs 72.9 ms) yet agree with each other far more closely than either agrees with a person, because they share a systematic bias. Human inter-annotator agreement on the same task is 27.0 ms. See the annotations release and the paper.

Provenance and licence

Fine-tuned from vasista22/whisper-hindi-large-v2 (Apache-2.0), itself derived from OpenAI Whisper large-v2 (MIT), on IndicVoices-R (CC BY 4.0). Released under Apache-2.0; please attribute all three.