yuriyvnv/whisper-large-v3-slovenian

Whisper-large-v3 Slovenian

Fine-tuned OpenAI Whisper-large-v3 for Slovenian automatic speech recognition, augmented with TTS-generated synthetic data.

This model is part of the paper: "Synthetic Speech Augmentation for Low-Resource Estonian and Slovenian ASR: Comparing Parakeet-TDT and Whisper" (Interspeech 2026). Paper coming soon.

Note: Fine-tuning Whisper on CommonVoice data degrades FLEURS performance for Slovenian. Zero-shot Whisper (37.02% raw / 19.18% normalized) outperforms all fine-tuned variants on FLEURS. This is because Whisper's pre-trained model handles FLEURS-style punctuation and casing, which fine-tuning on CV-style data erodes. On the in-domain CV17 test set, this model provides substantial improvements over both zero-shot and CV-only fine-tuning.

Model Description

• —Architecture: Encoder-decoder Transformer (Whisper)
• —Parameters: 1.55B
• —Tokenizer: 51,865-token byte-level BPE
• —Base model: openai/whisper-large-v3
• —Fine-tuning data: CommonVoice 17.0 Slovenian + ~5,850 synthetic sentences (paraphrase + domain expansion, without morphological category)
• —Training config: CV + Synth No Morph (synthetic corpus excluding morphological augmentation)

Evaluation Results

Raw WER/CER (no text normalization)

Normalized WER/CER (lowercase + punctuation removal)

Improvement over baselines

*FLEURS degrades vs. zero-shot due to domain mismatch (see note above), but improves substantially vs. CV-only fine-tuning.

All CV17 improvements are statistically significant (paired bootstrap, p < 0.001, n = 100,000).

Usage

from transformers import WhisperForConditionalGeneration, WhisperProcessor
import torch

# Load model and processor
model = WhisperForConditionalGeneration.from_pretrained("yuriyvnv/whisper-large-v3-slovenian")
processor = WhisperProcessor.from_pretrained("yuriyvnv/whisper-large-v3-slovenian")

# Load audio (16kHz mono)
import librosa
audio, sr = librosa.load("audio.wav", sr=16000)

# Transcribe
input_features = processor(audio, sampling_rate=16000, return_tensors="pt").input_features
with torch.no_grad():
    predicted_ids = model.generate(input_features, language="sl", task="transcribe")
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]
print(transcription)

Training Details

• —Optimizer: AdamW fused (lr=5e-5)
• —Schedule: Linear decay with 10% warmup
• —Effective batch size: 128 (64 x 2 gradient accumulation)
• —Epochs: 5
• —Best model: selected by eval_loss
• —Precision: bf16
• —Seed: 42

Synthetic Data Augmentation

The synthetic training data was generated using a three-stage pipeline:

1. 1.Text generation: GPT-5-mini generates diverse sentences across paraphrase, domain expansion, and morphological categories
2. 2.LLM-as-judge validation: Each sentence validated for grammaticality, naturalness, and language purity
3. 3.Speech synthesis: OpenAI gpt-4o-mini-tts with 11-voice rotation

Dataset: yuriyvnv/synthetic_asr_et_sl

Acknowledgments

• —Base model: OpenAI Whisper-large-v3
• —Training data: Mozilla Common Voice 17.0
• —Evaluation: Google FLEURS