awaaz-se-alfaaz/YouTube-Evaluation-Set

Awaaz se Alfaaz — YouTube Evaluation Set

This dataset is the realistic multi-speaker evaluation set used in Awaaz se Alfaaz, accepted at LaTeLL 2026 — "Enhancing Urdu ASR with Whisper v3: Fine-Tuning on Latest Datasets and Realistic Multi-Speaker Evaluation with SLM Post-Processing." It contains 30 short-form Urdu YouTube videos (YouTube Shorts) covering a mix of news, sports, and current affairs content, along with human annotated gold transcripts and transcripts produced by our fine-tuned model.

Dataset Details

• —Language: Urdu (ur)
• —Domain: Short-form YouTube news/current-affairs clips
• —Size: 30 audio clips
• —Task: Automatic Speech Recognition (ASR) evaluation

Dataset Structure

The dataset is provided as a single CSV file, awaaz_se_alfaaz_youtube_eval.csv, with the following columns:

Audio

Raw audio is not redistributed in this repository due to YouTube's terms of use. Instead, download_audio.py is provided to fetch the source audio for each clip using `yt-dlp`:

pip install yt-dlp
python download_audio.py

This downloads each video's audio as an .mp3 file into an audio/ folder, named by video_id, matching the video_id column in the CSV.

Dataset Creation

The 30 videos were sourced from Urdu-language YouTube channels covering news and current affairs. Gold transcripts were produced through manual annotation. Model transcripts were generated using our ASR pipeline: fine-tuned Whisper Large-v3-Turbo with LoRA and decoder optimization, followed by SLM-based post-processing with Qwen3-14B.

Licensing

This dataset (transcripts and metadata) is released under CC-BY-4.0. The underlying YouTube videos remain the property of their original creators/channels and are not redistributed as part of this dataset, only their IDs and links are provided, along with a script to fetch the audio for evaluation purposes.

Citation

If you use this dataset, please cite:

@inproceedings{awaazsealfaaz2026,
  title={Enhancing Urdu ASR with Whisper v3: Fine-Tuning on Latest Datasets and Realistic Multi-Speaker Evaluation with SLM Post-Processing},
  author={TODO: your names},
  booktitle={LaTeLL 2026},
  year={2026}
}