MLCommons/unsupervised_peoples_speech

Dataset Card for Unsupervised Peoples Speech

Table of Contents

• —Dataset Card for Unuspervised Peoples Speech
• —Table of Contents
• —Dataset Description
• —Dataset Summary
• —Dataset Structure
• —Relevant Statistics
• —Dataset Creation
• —Source Data
• —Annotations
• —Considerations for Using the Data
• —Additional Information
• —Licensing Information
• —Citation Information

Dataset Description

Dataset Summary

The Unsupervised Peoples Speech Dataset is a compilation of audiofiles extracted from Archive.org that is licensed for academic and commercial usage under CC-BY and CC-BY-SA licenses. It includes more than one million hours of audio with a diverse set of speakers.

• —Point of Contact: MLCommons Datasets Discord

Dataset Structure

This dataset is a collection of audio files that have been stored as tar files, each containing a set of audio files. On average, each tar file is 5GB in size.

• —All tar files are stored in either in the audio or audio2 directories.
• —The licenses.jsonl file contains the license information for each audio file.
• —The lang_id_results.jsonl file contains the predicted language for all files using Whisper Large V3.
• —The vad_results.jsonl file containes timestamps where voice was detected using Silero VAD.

Relevant Statistics

Duration Distribution

Most of the audios range between 1 and 10 minutes in length, with only 14 of them exceeding the 100 hour mark.

[image]

Sample Rates

99% of the audio in the dataset has a 44.1Khz sample rate, and the remaining audio varies from the more common 16Khz, 24Khz and 48 Khz to custom sample rates.

[image]

Dataset Creation

Source Data

Data was downloaded via the archive.org API. No data inference was done. No preprocessing was done.

Annotations

No manual annotation is done. We download only source audio. In particular, there is no "forced alignment" or "segmentation" done on this dataset.

Considerations for Using the Data

Our data is downloaded from archive.org. As such, the data is biased towards whatever users decide to upload there.

Almost all of our data is American accented English.

Additional Information

Licensing Information

The source data contains data under CC-BY-SA and CC-BY licenses. We license this dataset under https://creativecommons.org/licenses/by-sa/4.0/

Citation Information

Please cite

@article{USP,
author={Daniel Galvez and
          Ryan Hileman and
          Rafael Mosquera and
          Juan Ciro and
          Kurt Bollacker and
          Peter Mattson and
          David Kanter},
  title     = {Unsupervised People's Speech (The Million Hour Audio Dataset)},
  year      = {2023},
  url       = {https://huggingface.co/datasets/MLCommons/peoples_speech},
}