yuriyvnv/whisper-small-high-mixed-pt

Whisper-Small Portuguese - High-Quality Filtered Synthetic Data

This model is a fine-tuned version of openai/whisper-small for Portuguese automatic speech recognition (ASR). It was trained on Common Voice 17.0 Portuguese combined with WAVe-filtered high-quality synthetic speech data using a strict threshold (q ≥ 0.8).

Purpose

This model explores whether high-quality synthetic data filtering can overcome the limitations of smaller model architectures. The results reveal an important finding:

Key Finding: Even with strict quality filtering (q ≥ 0.8), the Small model shows no improvement over the CV-only baseline, demonstrating that the architectural capacity limitation cannot be overcome simply by improving synthetic data quality.

This provides evidence that model capacity, not data quality, is the limiting factor for smaller architectures.

Model Details

Evaluation Results

This Model (whisper-small-high-mixed-pt)

Comparison with Other Training Configurations (Whisper-Small Portuguese)

Key Performance Characteristics

• —Best cross-domain: Lowest MLS WER (30.40%) among all Small configurations
• —Marginal MLS improvement: Only 0.9% better than baseline on cross-domain
• —Worse in-domain: 14.28% vs 13.87% baseline (-3.0%)
• —Demonstrates capacity limitation: High-quality filtering doesn't overcome architectural constraints

Why High-Quality Filtering Doesn't Help Small Models

The paper explains this phenomenon:

"Compact models, with fewer parameters, struggle to disentangle the subtle acoustic differences between natural and synthetic speech. Unlike the Large-V3 model, which can exploit its deeper representational hierarchy to extract meaningful patterns, smaller models become overwhelmed by increased acoustic variability."

Contrast with Large-v3:

This 35+ percentage point difference demonstrates that the benefit of synthetic data is fundamentally tied to model capacity.

Training Data

Dataset Composition

WAVe Quality Distribution (Portuguese Synthetic Data)

Training Procedure

Hyperparameters

Training Infrastructure

• —GPU: NVIDIA H200 (140GB VRAM)
• —Operating System: Ubuntu 22.04
• —Framework: Hugging Face Transformers

Usage

Transcription Pipeline

from transformers import pipeline

transcriber = pipeline(
    "automatic-speech-recognition",
    model="yuriyvnv/whisper-small-high-mixed-pt",
    device="cuda"
)

result = transcriber("path/to/portuguese_audio.wav")
print(result["text"])

Direct Model Usage

from transformers import WhisperProcessor, WhisperForConditionalGeneration
import librosa

processor = WhisperProcessor.from_pretrained("yuriyvnv/whisper-small-high-mixed-pt")
model = WhisperForConditionalGeneration.from_pretrained("yuriyvnv/whisper-small-high-mixed-pt")
model.to("cuda")

audio, sr = librosa.load("path/to/portuguese_audio.wav", sr=16000)
input_features = processor(audio, sampling_rate=16000, return_tensors="pt").input_features.to("cuda")

predicted_ids = model.generate(input_features)
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]
print(transcription)

Specifying Language

model.generation_config.language = "pt"
model.generation_config.task = "transcribe"

When to Use This Model

This model is primarily useful for:

• —Research purposes: Demonstrating the impact of model capacity on synthetic data effectiveness
• —Slight cross-domain preference: Marginally better MLS performance (30.40% vs 30.69%)
• —Understanding architecture limitations: Comparing with Large-v3 results

For production use, consider:

• —whisper-small-cv-only-pt: Best Small model for Portuguese (13.87% WER)
• —whisper-large-v3-high-mixed-pt: Best accuracy (7.94% WER)

Research Implications

This model provides evidence for an important principle:

Synthetic data augmentation effectiveness scales with model capacity.

For practitioners:

• —Small models: Focus on high-quality real data; synthetic augmentation provides minimal benefit
• —Large models: Synthetic data with quality filtering dramatically improves performance
• —Resource planning: Don't invest in synthetic data generation for small model deployments

Limitations

• —Lower accuracy than baseline: 14.28% vs 13.87% (worse than CV-only)
• —Limited synthetic benefit: Architecture cannot leverage additional data effectively
• —Domain specificity: Optimized for general Portuguese
• —Dialect coverage: Performance may vary across Portuguese regional variants

Citation

This model is part of research on WAVe (Word-Aligned Verification) for synthetic speech quality assessment. While the WAVe methodology paper is currently under review, please cite our previous work that motivated this research:

@article{perezhohin2024enhancing,
  title={Enhancing Automatic Speech Recognition: Effects of Semantic Audio Filtering on Models Performance},
  author={Perezhohin, Yuriy and Santos, Tiago and Costa, Victor and Peres, Fernando and Castelli, Mauro},
  journal={IEEE Access},
  year={2024},
  publisher={IEEE}
}

@article{perezhohin2026wave,
  title={WAVe: Word-aligned verification of synthetic speech for ASR},
  author={Perezhohin, Yuriy and Castelli, Mauro},
  journal={Information Sciences},
  pages={123591},
  year={2026},
  publisher={Elsevier}
}

References

• —Base Model: openai/whisper-small
• —Training Data (Real): mozilla-foundation/common_voice_17_0
• —Training Data (Synthetic): yuriyvnv/synthetic_transcript_pt
• —Whisper Paper: Robust Speech Recognition via Large-Scale Weak Supervision
• —Motivating Research: Enhancing ASR with Semantic Audio Filtering (IEEE Access 2024)

License

Apache 2.0