RyanJJaison/kokborok-whisper-lora

Kokborok Whisper-small LoRA (research/demo checkpoint)

A LoRA adapter fine-tuning openai/whisper-small for automatic speech recognition on Kokborok (ISO 639-3 trp), a language spoken by roughly a million people in Tripura, India, with no prior first-party ASR support.

This is a research/demo checkpoint, not a production model. Test-set WER is above 1.0 (more edits than words in the reference), and the training corpus is a single speaker. See Limitations before using this for anything beyond experimentation. It is part of the Lahja project, a small tiered-fallback language layer for Kokborok (speech, text, translation, Q&A).

Model details

• —Base model: openai/whisper-small (244M params)
• —Adapter: LoRA, r=32, alpha=64, dropout=0.05, bias=none, target_modules=[q_proj, v_proj] — 3,538,944 trainable parameters (1.44% of the base model)
• —Language proxy: Whisper has no trp language token, so training and inference use bn (Bengali) as the decoder language. This is a script/tokenizer match, not a claim of phonetic similarity — Kokborok transcripts in this corpus are written in Bengali script, so the Bengali proxy gives the tokenizer a sane per-word subword count instead of falling back to raw bytes.
• —Framework: PEFT 0.20.0, transformers, trained with Seq2SeqTrainer

Training data

• —235 clips, 32.9 minutes total, mean clip length 8.1s (range 1.0–19.0s)
• —Single speaker — one narrator throughout, confirmed by cross-correlating the source recording's episode files against each other and by F0/spectral consistency across all clips
• —Source: a "Words of Life" Kokborok audio recording (18 short episodes, originally ~61 minutes), auto-captioned then filtered to keep only rows in clean Bengali script (captions that had drifted into Devanagari, Telugu, or stray Latin characters were dropped rather than trusted as ground truth)
• —Train/val/test are split by source episode, not by utterance — this corpus is one continuous recording, so a random utterance split would put adjacent seconds of the same sentence in both train and test. Episode-level splitting gives real topic/section holdout instead. 164 train / 36 val / 35 test utterances across 18 episode groups.

Why this matters for real-world use

Because every split shares the same speaker, reported WER/CER are speaker-optimistic: the model can partly learn this narrator's voice rather than only Kokborok in general. Expect materially worse performance on any other speaker. This is the single biggest caveat on these numbers — see Limitations.

Evaluation

Metrics from results/metrics.jsonl, scored with jiwer (corpus-level WER/CER, computed as total edits over total reference length, not averaged per utterance).

For reference, the untuned base model achieves 0 correct words out of 282 on the test set (every hypothesis is a repetition-loop artifact or a wrong-language guess); this adapter recovers real lexical content, cuts CER by roughly half, and eliminates the repetition loops — but WER above 1.0 means the model still inserts and substitutes more than it gets exactly right.

A note on checkpoint selection

This corpus overfits fast: validation loss bottoms out around epoch 7 and rises afterward, while validation WER (measured by a full generate pass, not loss) keeps improving to roughly epoch 14–15 before flattening. The checkpoint published here is the final epoch (20), not the validation-WER-selected checkpoint — if you are comparing against reports elsewhere for this project that cite WER 1.1348 (test) / 0.9288 (val), those numbers are for an earlier checkpoint (step 160) that scored best on validation WER at the time. Both checkpoints are close, unstably so, given the test set is only 35 utterances (a bootstrap confidence interval on numbers this size would be wide).

Limitations

• —Single-speaker training data. No speaker-disjoint evaluation exists for this model; do not expect these WER/CER numbers to hold for a speaker not in the training recording.
• —Test WER is above 1.0. This means, on average, there are more word insertions + substitutions + deletions than there are words in the reference — the model is not yet reliable enough for unsupervised use. Always have a human review output.
• —Small corpus. 32.9 minutes total, 2,239 word tokens with 75% appearing only once. The model has learned Kokborok's script and phonotactics far more than a wide vocabulary.
• —Bengali-script only. Kokborok is also written in Latin script in other contexts; this model was trained exclusively on Bengali-script transcripts and has not been evaluated on Romanized input.
• —`bn` language proxy. Decoder behavior (punctuation, casing conventions) inherits some biases from Bengali rather than Kokborok specifically.
• —One source recording. All training/validation/test audio derives from a single "Words of Life" programme; genre, topic, and recording conditions are narrow (devotional narration, one microphone, one room).

How to use

from transformers import WhisperForConditionalGeneration, WhisperProcessor
from peft import PeftModel
import librosa

base = WhisperForConditionalGeneration.from_pretrained("openai/whisper-small")
model = PeftModel.from_pretrained(base, "RyanJJaison/kokborok-whisper-lora")
processor = WhisperProcessor.from_pretrained(
    "RyanJJaison/kokborok-whisper-lora", language="bn", task="transcribe"
)

audio, sr = librosa.load("clip.wav", sr=16000, mono=True)
inputs = processor(audio, sampling_rate=sr, return_tensors="pt")
predicted_ids = model.generate(**inputs)
print(processor.batch_decode(predicted_ids, skip_special_tokens=True)[0])

Training procedure

• —20 epochs, effective batch size 16 (batch 4 × grad-accum 4), learning rate 1e-3, 8 warmup steps, bf16 precision
• —Trained on a single 6GB consumer GPU in ~16 minutes (952s)
• —Full training/eval code, config, and data pipeline: github.com/RyanJJaison/Lahja (asr/)

Framework versions

• —PEFT 0.20.0
• —transformers (see repository asr/requirements.txt for exact pin)

Citation

No formal citation; if this is useful, please link back to the Lahja repository.