kiviki/whisper-turbo-romani-lora

Whisper Turbo Romani LoRA

Status: experimental Whisper adapter baseline. This adapter is kept for reproducibility and comparison. In the current frozen local evaluation, zero-shot facebook/mms-1b-all with rmc-script_latin performs better on both evaluated splits, so this repository should not be read as the strongest current Romani ASR result.

This repository contains a PEFT LoRA adapter for openai/whisper-large-v3-turbo, fine-tuned for Latin-script Romani ASR.

This is not a standalone Whisper checkpoint. Load the base model from Hugging Face, then load this adapter and the tokenizer/processor files from this repository.

This package is a slim adapter export. It omits redundant full base embedding/projection matrices and keeps the LoRA weights plus the trained <|romani|> token deltas. Before loading the adapter, initialize the custom <|romani|> token from Whisper's Slovak token (<|sk|>), then force the Romani decoder prompt.

The training/evaluation harness that produced this adapter contains the required loader path:

python scripts/evaluate_hf_asr.py \
  --model-id openai/whisper-large-v3-turbo \
  --adapter-dir kiviki/whisper-turbo-romani-lora \
  --processor-dir kiviki/whisper-turbo-romani-lora \
  --language slovak \
  --task transcribe \
  --new-language-token romani \
  --max-length 72 \
  --no-repeat-ngram-size 3 \
  --repetition-penalty 1.15

Model Details

• —Base model: openai/whisper-large-v3-turbo
• —Adapter type: PEFT LoRA
• —LoRA rank: 16
• —LoRA alpha: 32
• —LoRA dropout: 0.05
• —Target modules: q_proj, v_proj
• —Encoder: frozen during training
• —Custom Whisper language token: <|romani|>
• —Decoder prompt: <|romani|><|transcribe|><|notimestamps|>

Frozen Evaluation

All results below were rerun on 2026-08-10 with fixed manifests and guarded decoding. Lower WER/CER is better.

Naming used below:

• —Prompt Test = test.csv, the original single-source prompt set (R_prompts).
• —Multi-Recording Dev = validation_clean.csv, the cleaned multi-source validation split.
• —Whisper Turbo Baseline = base Whisper with a Slovak transcription prompt and no Romani fine-tuning.
• —Whisper Turbo Romani LoRA = this fine-tuned adapter.
• —MMS-1B-All Romani/Latin Zero-Shot = public facebook/mms-1b-all with target_lang=rmc-script_latin, with no local MMS training.

Prompt Test contains one source group only, so it is useful for continuity with the original model card but too easy to be the only headline result. Multi-Recording Dev spans five recording groups and is a better stress test for iteration.

The detailed frozen report is included in this repository at eval/frozen-asr-evaluation-2026-08-10.md.

Known Limitations

This is an experimental low-resource ASR adapter. It still makes frequent word-boundary, spelling, and diacritic mistakes. It is kept as a reproducible Whisper adapter baseline while MMS adapter training is investigated separately.

The MMS rows above are comparison results only. This repository does not contain an MMS adapter.

Intended Use

Use this adapter for experimentation with Latin-script Romani transcription. Review transcripts before using them in user-facing or high-stakes settings.

Not Included

This repository does not include the base Whisper model, raw audio data, training optimizer state, or training/evaluation logs.