espnet/Bagpiper_PreTrain_Data

Bagpiper Pretraining Data

Bagpiper Pretraining Data is the public rich-captioned audio snapshot associated with Bagpiper, an open-ended audio language model that learns bidirectional mappings between audio and comprehensive text descriptions across speech, music, environmental sound, and mixtures.

The en metadata describes the primary rich-caption language. Source audio can contain speech or singing in other languages; it is not an English-only audio guarantee.

The repository contains 155,151,789 rows in 7,780 valid Parquet shards across 18 source-family directories, using 5.229 TB. It is intended for large-scale research workflows; select only the source families needed for your experiment rather than downloading the entire repository by default.

Repository snapshot

The clotho_train directory also contains 43 zero-row, schema-only .parquet.tmp files; they are temporary artifacts, are excluded from all counts above, and must not be loaded. These are physical repository statistics for revision 1997390aef538950729203ed4886b78b077d1a71. Directory names identify packing families and do not themselves establish ownership or a uniform license.

Loading selected Parquet families

Use explicit Parquet globs to keep data acquisition intentional:

from datasets import load_dataset

dataset = load_dataset(
    "parquet",
    data_files={
        "train": [
            "hf://datasets/espnet/Bagpiper_PreTrain_Data/audiocaps/*.parquet",
            "hf://datasets/espnet/Bagpiper_PreTrain_Data/clotho_train/*.parquet",
        ]
    },
    split="train",
    streaming=True,
)

print(dataset.features)
example = next(iter(dataset))

All audited valid shards use one shared schema:

direction denotes whether the pair is eligible for audio-to-text understanding, text-to-audio generation, or both; it does not imply duplicated physical rows. Inspect dataset.features before writing a consumer and ignore .parquet.tmp files.

The Hub datasets-server currently exposes only a default clotho_train view (29,434 rows), not the complete 18-family snapshot. Use the explicit glob approach above for the intended families.

Relationship to the paper

The Bagpiper paper describes a broader pretraining pipeline beginning from approximately 422 million raw audio-caption pairs, with clips capped at 30 seconds and captions generated by a Qwen3-Omni captioner. Its 600B figure is a training-token budget—300B text-to-audio, 150B audio-to-text, and 150B text-only—not the row count or byte size of this Hub snapshot.

This repository is a public artifact associated with that pipeline. Do not infer that its 18 directories are a complete one-to-one dump of every raw pair or every text-only source described in the paper.

Construction overview

Audio from heterogeneous speech, music, and sound collections is paired with rich captions generated by the paper's Qwen3-Omni-30B-A3B-Captioner. A Qwen3-32B classifier separates speech, music, and sound; text/audio quality and alignment filtering uses heuristic and LLM judgments, UTMOS for speech, AudioBox-Aesthetics for non-speech, CLAP alignment, Gumbel top-k sampling, and MinHash text deduplication. Rich captions can describe:

• —speech transcription, speaker attributes, language/accent, timing, and acoustic conditions;
• —instruments, genre, rhythm, melody, harmony, and musical structure;
• —environmental events, temporal ordering, spatial context, ambience, and recording quality.

Bagpiper uses these descriptions in both directions: audio-to-rich-caption for understanding and rich-caption-to-audio for generation, alongside text-only language-model data.

Limitations, provenance, and responsible use

• —Captions are machine-generated and may hallucinate content, timing, speaker attributes, or acoustic details.
• —Source families are heterogeneous and can include web-derived or transformed media, copyrighted speech/music, identifiable voices, and sensitive content.
• —This repository does not declare a blanket license. Users must review the terms, attribution requirements, privacy/consent constraints, and redistribution permissions of the underlying source represented by each row before use or further redistribution.
• —Directory-level public availability is not proof that every underlying asset is cleared for every commercial or biometric use.
• —The corpus is not exhaustively moderated for personal information, unsafe content, bias, or offensive language.
• —Generated captions and source media can inherit demographic, linguistic, geographic, and cultural biases.

For provenance corrections or takedown requests, use the repository community tab and provide the source-family directory plus a stable row identifier. Do not repost sensitive media in the report.

Related resources

• —Paper: Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions
• —Project: bagpiper-cmu.github.io
• —Base model: espnet/bagpiper
• —Bagpiper SFT data: espnet/Bagpiper_SFT_Data
• —Bagpiper-TTS SFT data: espnet/Bagpiper_TTS_SFT_Data
• —ESPnet: espnet/espnet

Citation

@inproceedings{tian2026bagpiper,
  title={Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions},
  author={Tian, Jinchuan and Wang, Haoran and Su, Bo-Hao and Huang, Chien-yu and Wang, Qingzheng and Shi, Jiatong and Chen, William and Gong, Xun and Arora, Siddhant and Li, Chin-Jou and Someki, Masao and Maekaku, Takashi and Goto, Keita and Shinohara, Yusuke and Sakuma, Jin and Yang, Chao-Han Huck and Watanabe, Shinji},
  booktitle={Third Conference on Language Modeling},
  year={2026},
  url={https://openreview.net/forum?id=FuHs64E3X6}
}