x2q/whisper-large-v3-da-coral-lora

whisper-large-v3-da-coral-lora

LoRA fine-tune of openai/whisper-large-v3 for Danish speech recognition, trained on CoRal-project/coral-v3 (conversation split) with telephone-codec augmentation to close the domain gap between CoRal's clean studio recordings and real telephone-call audio.

Model description

• —Base model: openai/whisper-large-v3
• —Method: LoRA (PEFT), rank=32, alpha=64, dropout=0.05, target modules: q_proj, v_proj
• —Trainable parameters: 15.7M (1.01% of the 1.56B total)

Training data

CoRal-project/coral-v3 — real, human-transcribed Danish speech collected by the Alexandra Institute with Danish Innovation Fund support. Only the conversation split was used (not read_aloud); ~10,000 raw samples were filtered and augmented down to 9,027 training chunks (~9.9 hours of audio):

• —~74% telephone-codec-simulated (8kHz bandwidth limiting, low-bitrate mp3 round-trip, light noise) — to match real telephone-call audio, since coral-v3's raw recordings are clean/studio-quality, not phone-quality
• —~26% clean (loudness-normalized only) — to help retain general Danish transcription ability alongside the telephone-domain adaptation
• —All samples loudness-normalized (EBU R128, -16 LUFS) — coral-v3's raw recordings have highly inconsistent levels, some near-silent

coral-v3 is released under an OpenRAIL license; this model inherits that license.

Training procedure

• —2 epochs, effective batch size 32 (per-device batch 1, grad accumulation 32)
• —Learning rate 1e-4, linear warmup over the first 40 of 548 total steps
• —fp16, gradient checkpointing
• —No read_aloud data was used in training (see evaluation below for the effect)

Evaluation

Evaluated against a held-out coral-v3 set (94 conversation + 100 read_aloud samples, disjoint from training — guaranteed by skipping the exact sample count training consumed with the same shuffle seed) with real ground truth, so WER is directly computable:

This model nearly halves conversational Danish WER relative to the untrained base, closing most of the gap to CoRal's own purpose-built model starting from a generic base. The read_aloud improvement is smaller because training only used the conversation split.

Limitations

A qualitative audit on 30 real (non-coral, actual telephone call) Danish recordings found a ~13% fabrication rate — invented person names, invented entities, or meaning-inverting errors not present in the base model's output for the same audio. Examples observed: inventing a name not supported by any reference transcription, and inverting a stated success/failure outcome. The untrained base model has a comparable overall failure rate on real calls too (it has its own well-known hallucination pattern on quiet audio, which this model appears to suppress, at the cost of introducing different errors elsewhere) — so this isn't unique to fine-tuning, but it does mean:

• —Do not use this model as a sole source for automated name/entity extraction without cross-checking against a second model or human review.
• —Output drops most punctuation and capitalization, following coral-v3's transcription convention — apply punctuation restoration as a post-process step if a clean, readable transcript is required.
• —Performs noticeably better on informal conversational Danish than on formal/read-aloud style speech, since training only covered the former.

Usage

import torch
from transformers import WhisperForConditionalGeneration, WhisperProcessor
from peft import PeftModel

REPO = "x2q/whisper-large-v3-da-coral-lora"
processor = WhisperProcessor.from_pretrained(REPO, language="danish", task="transcribe")
base = WhisperForConditionalGeneration.from_pretrained(
    "openai/whisper-large-v3", torch_dtype=torch.float16
).to("cuda")
model = PeftModel.from_pretrained(base, REPO).eval()

# inputs = processor.feature_extractor(audio, sampling_rate=16000, return_tensors="pt") \
#     .input_features.to("cuda").half()
# ids = model.generate(inputs, language="danish", task="transcribe", num_beams=5)
# processor.tokenizer.decode(ids[0], skip_special_tokens=True)