Parakeet-Inc/J-HARD-TTS-Eval

J-HARD-TTS-Eval

 

[!NOTE] For full documentation, detailed benchmark results, and methodology, please refer to the **GitHub Repository**.

Overview

J-HARD-TTS-Eval is a benchmark designed to evaluate the robustness of autoregressive Japanese Text-To-Speech (TTS) models. It focuses on specific failure modes such as stability in short sequences, repetition handling, and context completion.

Usage

You can easily load the dataset using the Hugging Face datasets library.

from datasets import load_dataset

# Load a specific subset: "short", "repetition", "rhyme", or "continuation"
dataset = load_dataset("Parakeet-Inc/J-HARD-TTS-Eval", "rhyme", split="test")

sample = dataset[0]

# Accessing data
print(f"ID: {sample['id']}")
print(f"Target Text: {sample['text']}")
print(f"Prompt Text: {sample['prompt_text']}")
print(f"Prompt Audio: {sample['prompt_audio']['array'].shape}, Sampling Rate: {sample['prompt_audio']['sampling_rate']}")

# Play audio in a notebook
# import IPython.display as ipd
# ipd.Audio(sample["prompt_audio"]["array"], rate=sample["prompt_audio"]["sampling_rate"])

Dataset Structure

• —id (string): Unique identifier for the sample (e.g., rhyme-01).
• —text (string): The target text to be synthesized (Ground Truth Text).
• —prompt_text (string): The transcript of the prompt audio.
• —prompt_audio (Audio): The reference audio used for zero-shot prompting (sourced from Common Voice).

Citation

If you use this benchmark in your research, please cite our paper:

@inproceedings{imai2025jhard,
  author    = {Imai, Shuhei and Enomoto, Haruhisa and Kaneko, Takeshi and Nakamura, Taiki},
  title     = {A Study on a High-Difficulty Japanese Text-to-Speech Corpus Specialized for Sequence Consistency Evaluation},
  booktitle = {Proc. ASJ},
  year      = {2025},
  month     = {9}
}