Datasetpublic

FluidInference/cv-corpus-25.0-ja

Mozilla Common Voice 25.0 - Japanese Test Set (Complete) Dataset Description Complete Japanese test set from Mozilla Common Voice Corpus 25.0. This dataset contains all 9,019 validated test samples, compared to the partial 2,334-sample version previously available on HuggingFace. Key Features Size: 9,019 validated test utterances Coverage: 100% of official Common Voice 25.0 Japanese test split Multi-speaker: Diverse set of speakers with demographic… See the full description on the dataset page: https://huggingface.co/datasets/FluidInference/cv-corpus-25.0-ja.

sourceHugging Facecc0-1.0updated 6mo agoView on Hugging Face

0likes162downloads

Dataset Card

Mozilla Common Voice 25.0 - Japanese Test Set (Complete)

Dataset Description

Complete Japanese test set from Mozilla Common Voice Corpus 25.0. This dataset contains all 9,019 validated test samples, compared to the partial 2,334-sample version previously available on HuggingFace.

Key Features

—Size: 9,019 validated test utterances
—Coverage: 100% of official Common Voice 25.0 Japanese test split
—Multi-speaker: Diverse set of speakers with demographic metadata
—Quality: Community-validated recordings
—Format: MP3 audio files with full metadata
—Use case: Standard test set for Japanese ASR evaluation

Why This Dataset?

The previous HuggingFace repository (FluidInference/cv-corpus-25.0-ja) only contained 2,334 test files (26%) due to incomplete uploads. This dataset provides:

—✅ All 9,019 test files (100%)
—✅ Complete metadata matching official Mozilla release
—✅ Ready-to-use format for ASR benchmarking
—✅ No missing files or metadata mismatches

Dataset Structure

Files

cv-corpus-25.0-ja-test-only/
├── manifest.json          # Dataset manifest with split information
├── load_dataset.py        # Helper script to load all splits
├── ja_00/
│   ├── clips/             # 3,000 MP3 files
│   └── test.jsonl         # Metadata for this split
├── ja_01/
│   ├── clips/             # 3,000 MP3 files
│   └── test.jsonl         # Metadata for this split
├── ja_02/
│   ├── clips/             # 3,000 MP3 files
│   └── test.jsonl         # Metadata for this split
└── ja_03/
    ├── clips/             # 19 MP3 files
    └── test.jsonl         # Metadata for this split

Metadata Format

Each test.jsonl file contains entries like:

json

{
  "file_name": "common_voice_ja_12345.mp3",
  "path": "clips/common_voice_ja_12345.mp3",
  "text": "Japanese transcription text",
  "client_id": "anonymous_speaker_id",
  "up_votes": 2,
  "down_votes": 0,
  "age": "thirties",
  "gender": "male",
  "accent": "",
  "locale": "ja"
}

Note: Files are split across 4 directories (ja00, ja01, ja02, ja03) to comply with HuggingFace's 10,000 files per directory limit. Each directory contains a subset of the full test set.

Dataset Manifest

The manifest.json file provides complete information about the dataset structure:

json

{
  "dataset_name": "Mozilla Common Voice 25.0 - Japanese Test Set",
  "total_samples": 9019,
  "total_size_mb": 247.16,
  "num_splits": 4,
  "splits": [
    {"name": "ja_00", "num_files": 3000, ...},
    {"name": "ja_01", "num_files": 3000, ...},
    {"name": "ja_02", "num_files": 3000, ...},
    {"name": "ja_03", "num_files": 19, ...}
  ]
}

Usage

Loading with Python

python

import json
from pathlib import Path

def load_cv_test_set(dataset_dir="."):
    dataset_dir = Path(dataset_dir)
    samples = []

    # Load from all splits (ja_00, ja_01, ja_02, ja_03)
    for split_dir in sorted(dataset_dir.glob("ja_[0-9][0-9]")):
        metadata_file = split_dir / "test.jsonl"

        with open(metadata_file, 'r', encoding='utf-8') as f:
            for line in f:
                entry = json.loads(line)
                entry['audio_path'] = str(split_dir / entry['path'])
                samples.append(entry)

    return samples

# Load complete test set
test_samples = load_cv_test_set()
print(f"Loaded {len(test_samples)} test samples")

ASR Benchmarking

python

# Evaluate your ASR model
for sample in test_samples:
    audio_path = sample['audio_path']
    reference = sample['text']

    # Your ASR inference here
    hypothesis = your_asr_model(audio_path)

    # Calculate CER/WER
    cer = calculate_cer(reference, hypothesis)

Dataset Statistics

—Total samples: 9,019
—Language: Japanese (ja)
—Format: MP3 (various bitrates)
—License: CC0 1.0 (Public Domain)
—Source: Mozilla Common Voice 25.0
—Split: Test only

Comparison with Other Datasets

Dataset	Samples	Completeness
FluidInference/cv-corpus-25.0-ja	2,334	26%
This dataset	9,019	100%
Original Mozilla CV 25.0 test	9,019	100%

Citation

If you use this dataset, please cite the original Common Voice project:

bibtex

@inproceedings{commonvoice:2020,
  author = {Ardila, R. and Branson, M. and Davis, K. and Henretty, M. and Kohler, M. and Meyer, J. and Morais, R. and Saunders, L. and Tyers, F. M. and Weber, G.},
  title = {Common Voice: A Massively-Multilingual Speech Corpus},
  booktitle = {Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020)},
  pages = {4211--4215},
  year = {2020}
}

License

CC0 1.0 Universal (Public Domain)

The Common Voice dataset is released under CC0, meaning you can:

—Use for any purpose (commercial or non-commercial)
—Modify and redistribute
—No attribution required (though appreciated)

Original Source

—Project: Mozilla Common Voice
—Version: 25.0 (released 2026-03-09)
—Language: Japanese (ja)
—Original download: https://commonvoice.mozilla.org/ja/datasets

Dataset Quality

All samples in this dataset have been:

—✅ Validated by community members
—✅ Checked for audio quality
—✅ Verified for transcription accuracy
—✅ Filtered from invalidated/reported samples

Use Cases

—Japanese ASR model evaluation
—Benchmarking speech recognition systems
—Speaker diversity analysis
—Accent and demographic studies
—Standard test set for research papers

Acknowledgments

Thanks to the Mozilla Common Voice community and all the contributors who recorded and validated these audio samples.