Datasetpublic

MihaiPopa-1/common_voice_22_0_toki_pona_parquet

Common Voice 22.0 - Toki Pona Subset! My own Parquet conversion of Toki Pona's subset of Fsicoli's reupload of Common Voice 22 so we don't have to downgrade to Datasets 3.6 anymore! Why? Because the original dataset required Hugging Face Datasets 3.6 or older because it has Python code and it's in TAR shards. This is in Parquet and works with any recent version of Hugging Face Datasets! Details Dataset Structure DatasetDict({… See the full description on the dataset page: https://huggingface.co/datasets/MihaiPopa-1/common_voice_22_0_toki_pona_parquet.

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face

0likes26downloads

Dataset Card

Common Voice 22.0 - Toki Pona Subset!

My own Parquet conversion of Toki Pona's subset of Fsicoli's reupload of Common Voice 22 so we don't have to downgrade to Datasets 3.6 anymore!

Why?

Because the original dataset required Hugging Face Datasets 3.6 or older because it has Python code and it's in TAR shards.

This is in Parquet and works with any recent version of Hugging Face Datasets!

Details

Dataset Structure

DatasetDict({
    train: Dataset({
        features: ['client_id', 'path', 'audio', 'sentence', 'up_votes', 'down_votes', 'age', 'gender', 'accent', 'locale', 'segment', 'variant'],
        num_rows: 2630
    })
    validation: Dataset({
        features: ['client_id', 'path', 'audio', 'sentence', 'up_votes', 'down_votes', 'age', 'gender', 'accent', 'locale', 'segment', 'variant'],
        num_rows: 2217
    })
    test: Dataset({
        features: ['client_id', 'path', 'audio', 'sentence', 'up_votes', 'down_votes', 'age', 'gender', 'accent', 'locale', 'segment', 'variant'],
        num_rows: 2166
    })
    other: Dataset({
        features: ['client_id', 'path', 'audio', 'sentence', 'up_votes', 'down_votes', 'age', 'gender', 'accent', 'locale', 'segment', 'variant'],
        num_rows: 4997
    })
    invalidated: Dataset({
        features: ['client_id', 'path', 'audio', 'sentence', 'up_votes', 'down_votes', 'age', 'gender', 'accent', 'locale', 'segment', 'variant'],
        num_rows: 358
    })
})

Duration in Hours

This excludes the invalidated set from this calculation!

train: 2.55 hours
validation: 2.61 hours
test: 2.50 hours
other: 7.14 hours
TOTAL: 14.80 hours

Code to Reproduce

By GPT-5.6 Luna and Gemini 2.5 Flash:

python

pip install "datasets>=3,<4" huggingface_hub

from datasets import load_dataset

ds = load_dataset(
    "fsicoli/common_voice_22_0",
    "tok",
    trust_remote_code=True,
)

print(ds)

Then:

python

import os

def get_duration_batched(examples):
    durations = []
    for audio_item in examples["audio"]:
        # Calculating duration still requires accessing the audio array to get its length
        durations.append(len(audio_item["array"]) / audio_item["sampling_rate"])
    examples["duration"] = durations
    return examples

hours = 0

for split in ["train", "validation", "test", "other"]:
    # Use batched=True to process multiple examples at once
    # Use remove_columns=["audio"] to free up memory after duration is calculated
    # Use num_proc for parallel processing (adjust based on your system's cores and memory)
    tmp = ds[split].map(
        get_duration_batched,
        batched=True,
        remove_columns=["audio"],
        num_proc=os.cpu_count() # Consider reducing if memory remains an issue
    )
    split_hours = sum(tmp["duration"]) / 3600

    print(f"{split}: {split_hours:.2f} hours")
    hours += split_hours

print(f"\nTOTAL: {hours:.2f} hours")

Extra Info

License (this repo): Apache 2.0

License (original dataset and Fsicoli's reupload): CC0 (Public Domain)