CoolFace
Datasetpublic

norbertm/whisper-eval-rare-languages-csv

Whisper 3 Large Evaluation on Mozilla Common Voice 17 Rare Languages (Enhanced Metrics) Dataset Description This enhanced dataset contains comprehensive evaluation results of OpenAI's Whisper 3 Large model on rare languages from Mozilla Common Voice 17, with extensive additional metrics for thorough ASR evaluation. Key Features Enhanced Error Metrics: WER (Word Error Rate): Standard word-level error measurement CER (Character Error Rate):… See the full description on the dataset page: https://huggingface.co/datasets/norbertm/whisper-eval-rare-languages-csv.

sourceHugging Facemitupdated 1y agoView on Hugging Face
0likes33downloads
Dataset Card

Whisper 3 Large Evaluation on Mozilla Common Voice 17 Rare Languages (Enhanced Metrics)

Dataset Description

This enhanced dataset contains comprehensive evaluation results of OpenAI's Whisper 3 Large model on rare languages from Mozilla Common Voice 17, with extensive additional metrics for thorough ASR evaluation.

Key Features

Enhanced Error Metrics:

  • —WER (Word Error Rate): Standard word-level error measurement
  • —CER (Character Error Rate): Character-level error measurement
  • —MER (Match Error Rate): Alternative error rate calculation
  • —WIL (Word Information Lost): Information loss measurement

Edit Distance Analysis:

  • —Word-level and character-level edit distances
  • —Normalized edit distance metrics
  • —Comprehensive distance analysis

Length and Structure Metrics:

  • —Word, character, and sentence counts
  • —Length ratios and differences
  • —Average word length analysis
  • —Sentence structure preservation

Script-Specific Analysis:

  • —Latin, Cyrillic, Armenian, Georgian, Tamil, Bengali character ratios
  • —Punctuation preservation analysis
  • —Script-specific performance metrics

Statistical Metrics:

  • —Jaccard similarity for vocabulary overlap
  • —Frequency correlation analysis
  • —Vocabulary union and overlap metrics
  • —Unique word analysis

Dataset Statistics

  • —Total samples: 111,507
  • —Languages: 21 rare languages
  • —Total metrics: 56 comprehensive evaluation metrics
  • —Scripts covered: Latin, Cyrillic, Armenian, Georgian, Tamil, Bengali

Language Coverage

LanguageCodeScriptSample Count
AssameseasBengali~551
BretonbrLatin~2,212
WelshcyLatin~5,379
EstonianetLatin~2,653
BasqueeuLatin~13,630
GalicianglLatin~9,990
HungarianhuLatin~11,435
ArmenianhyArmenian~4,281
GeorgiankaGeorgian~12,618
KazakhkkCyrillic~514
LithuanianltLatin~4,753
LatvianlvLatin~6,752
MacedonianmkCyrillic~1,097
MaltesemtLatin~1,662
OccitanocLatin~254
SlovakskLatin~5,000
SlovenianslLatin~1,242
SwahiliswLatin~12,253
TamiltaTamil~12,074
TurkmentkLatin~546
TatarttCyrillic~4,964

Performance Highlights

Top Performing Languages (by WER):

  1. 1.Hungarian (hu): WER = 0.1822
  2. 2.Galician (gl): WER = 0.2027
  3. 3.Slovenian (sl): WER = 0.2205
  4. 4.Macedonian (mk): WER = 0.2762
  5. 5.Latvian (lv): WER = 0.3021

Usage

python
from datasets import load_dataset

# Load the enhanced dataset
dataset = load_dataset("norbertm/whisper-eval-rare-languages-csv")

# Access comprehensive metrics
print(dataset['train'][0])

Research Applications

This enhanced dataset enables:

  1. 1.Comprehensive ASR Evaluation: Multiple error metrics for thorough analysis
  2. 2.Script-Specific Analysis: Understanding performance across different writing systems
  3. 3.Statistical Analysis: Vocabulary and frequency correlation studies
  4. 4.Length Analysis: Understanding how text length affects recognition
  5. 5.Cross-Language Comparison: Detailed performance comparison across 21 languages

Citation

If you use this dataset in your research, please cite:

bibtex
@dataset{whisper_eval_enhanced_2024,
  title={Whisper 3 Large Evaluation on Mozilla Common Voice 17 Rare Languages (Enhanced Metrics)},
  author={norbertm},
  year={2024},
  publisher={Hugging Face},
  url={https://huggingface.co/datasets/norbertm/whisper-eval-rare-languages-csv}
}

License

This dataset is licensed under the MIT License.

This enhanced version includes 46 additional metrics beyond the original WER and CER, providing unprecedented depth for ASR evaluation research.