retkowski/ytseg

YTSeg: A Benchmark for Audio Chaptering and Video Transcript Segmentation

We present <span style="font-variant:small-caps; font-weight:700;">YTSeg</span>, a topically and structurally diverse benchmark for the audio chaptering and transcript segmentation task based on YouTube videos. The dataset comprises 19,299 videos from 393 channels, amounting to 6,533 content hours. The topics are wide-ranging, covering domains such as science, lifestyle, politics, health, economy, and technology. The videos are from various types of content formats, such as podcasts, lectures, news, corporate events \& promotional content, and, more broadly, videos from individual content creators. The benchmark was introduced in *From Text Segmentation to Smart Chaptering: A Novel Benchmark for Structuring Video Transcriptions ([acl](https://aclanthology.org/2024.eacl-long.25/) | [arXiv](https://arxiv.org/abs/2402.17633)). A follow-up paper, Beyond Transcripts: A Renewed Perspective on Audio Chaptering* (acl | arXiv), extends it with additional ASR transcripts, duration and speaker annotations, and embedded audio data. We provide both text and audio data as well as a download script for the video data.

Data Overview

We offer three dataset subsets:

• —Text — For text-based segmentation and chaptering approaches using transcripts.
• —Audio — For audio-based chaptering approaches with embedded audio.
• —Titles — For chapter title generation given segment text (relevant for two-stage approaches).

<span style="font-variant:small-caps;">YTSeg</span> (Text)

Each video is represented as a JSON object. The fields are organized into three categories: Transcripts, Target Representations, and Metadata.

Transcripts

We provide three transcript variants for each video: the original reference transcript and two ASR-generated transcripts using Whisper models.

Target Representations

Multiple target formats are provided for different modeling approaches.

Metadata

Partition Statistics

<span style="font-variant:small-caps;">YTSeg</span> (Audio)

The audio config provides the complete dataset with embedded audio files. Each video is represented with the same fields as the text config, plus an audio field containing the preprocessed audio data.

Audio

All other fields (transcripts, target representations, and metadata) are identical to the Text config described above.

Partition Statistics

<span style="font-variant:small-caps;">YTSeg</span> (Titles)

Each chapter of a video is represented as a JSON object with the following fields:

Video Data

A download script for the video and audio data is provided.

python download_videos.py

In the script, you can further specify a target folder (default is ./video) and target formats in a priority list.

Loading Data

The dataset can be loaded directly using the HuggingFace datasets library:

from datasets import load_dataset

# Load the audio config (with embedded audio)
dataset = load_dataset("retkowski/ytseg", "audio", split="test")

# Load the text config (text-only)
dataset = load_dataset("retkowski/ytseg", "text", split="test")

# Load the titles config
dataset = load_dataset("retkowski/ytseg", "titles", split="test")

Note on Binary Labels: The binary segmentation labels (e.g., target_binary_ref) are prefixed with |= to force the field to be stored as a string, preventing leading zeros from being lost during processing. For actual usage, strip the |= prefix:

binary_labels = dataset['target_binary_ref'].lstrip('|=')

Citing

We kindly request you to cite our corresponding papers if you use our dataset.

@inproceedings{retkowski-waibel-2024-text,
    title = "From Text Segmentation to Smart Chaptering: A Novel Benchmark for Structuring Video Transcriptions",
    author = "Retkowski, Fabian  and Waibel, Alexander",
    editor = "Graham, Yvette  and Purver, Matthew",
    booktitle = "Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = mar,
    year = "2024",
    address = "St. Julian{'}s, Malta",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.eacl-long.25",
    pages = "406--419",
    abstract = "Text segmentation is a fundamental task in natural language processing, where documents are split into contiguous sections. However, prior research in this area has been constrained by limited datasets, which are either small in scale, synthesized, or only contain well-structured documents. In this paper, we address these limitations by introducing a novel benchmark YTSeg focusing on spoken content that is inherently more unstructured and both topically and structurally diverse. As part of this work, we introduce an efficient hierarchical segmentation model MiniSeg, that outperforms state-of-the-art baselines. Lastly, we expand the notion of text segmentation to a more practical {``}smart chaptering{''} task that involves the segmentation of unstructured content, the generation of meaningful segment titles, and a potential real-time application of the models.",
}

@inproceedings{retkowski-etal-2026-beyond,
    title = "Beyond Transcripts: A Renewed Perspective on Audio Chaptering",
    author = {Retkowski, Fabian  and
      Z{\"u}fle, Maike  and
      Nguyen, Thai Binh  and
      Niehues, Jan  and
      Waibel, Alexander},
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.acl-long.396/",
    doi = "10.18653/v1/2026.acl-long.396",
    pages = "8765--8787",
    ISBN = "979-8-89176-390-6",
    abstract = "Audio chaptering, the task of automatically segmenting long-form audio into coherent sections, is increasingly important for navigating podcasts, lectures, and videos. Despite its relevance, research remains limited and text-based, leaving key questions unresolved about leveraging audio information, handling ASR errors, and transcript-free evaluation. We address these gaps through three contributions: (1) a systematic comparison between text-based models with acoustic features, a novel audio-only architecture (AudioSeg) operating on learned audio representations, and multimodal LLMs; (2) empirical analysis of factors affecting performance, including transcript quality, acoustic features, duration, and speaker composition; and (3) formalized evaluation protocols contrasting transcript-dependent text-space protocols with transcript-invariant time-space protocols. Our experiments on YTSeg reveal that AudioSeg substantially outperforms text-based approaches, pauses provide the largest acoustic gains, and current MLLMs struggle due to context limitations and weak instruction following."
}

Changelog

• —20.01.2026 -- Major data and format update:
• —Added ASR transcripts (Whisper-tiny and Whisper-large), structured transcript targets with timestamps, and metadata for finer-grained analysis (speaker category, dominant speaker proportion, number of speakers, duration)
• —Added audio config with HuggingFace Audio feature for seamless loading with embedded audio
• —Updated to use HuggingFace datasets library for data loading (replacing local pandas scripts and use proper HF configs)
• —Updated YTSeg[Titles] field names for clarity
• —25.07.2024 -- Added complete list of chapter titles to YTSeg (YTSeg[Titles] is a filtered subset)
• —09.04.2024 -- Added audio data
• —27.02.2024 -- Initial release

License

The dataset is available under the Creative Commons Attribution-NonCommercial-ShareAlike (CC BY-NC-SA) 4.0 license. We note that we do not own the copyright of the videos and as such opted to release the dataset with a non-commercial license, with the intended use to be in research and education.