AlienKevin/sbs_cantonese

SBS Cantonese Speech Corpus

This speech corpus contains 435 hours of SBS Cantonese podcasts from Auguest 2022 to October 2023. There are 2,519 episodes and each episode is split into segments that are at most 10 seconds long. In total, there are 189,216 segments in this corpus. Here is a breakdown on the categories of episodes present in this dataset:

<style> table th:first-of-type { width: 5%; } table th:nth-of-type(2) { width: 15%; } table th:nth-of-type(3) { width: 50%; } </style>

• —Uncategorized episodes are mostly news but also contains other categories listed above.

Dataset Details

Dataset Description

<!-- Provide a longer summary of what this dataset is. -->

• —Curated by: Kevin Li
• —Language(s): Cantonese, English (only in podcasts categorized as "english")
• —License: Creative Commons Attribution Non-Commercial 4.0

Scraper

<!-- Provide the basic links for the dataset. -->

• —Repository: https://github.com/AlienKevin/sbs_cantonese

Uses

Each episode is split into segments using silero-vad. Since silero-vad is not trained on Cantonese data, the segmentation is not ideal and often break sentences in the middle. Hence, this dataset is not intended to be used for supervised ASR. Instead, it is intended to be used for self-supervised speech pretraining, like training WavLM, HuBERT, and Wav2Vec.

Format

Each segment is stored as a monochannel FLAC file with a sample rate of 16k Hz. You can find the segments under the audio/ folder, where groups of segments are bundled into a .tar.gz file for ease of distribution.

The filename of the segment shows which episodes it belongs to and place of it within that episode: For example, here's a filename:

0061gy0w8_0000_5664_81376

where

• —0061gy0w8 is the episode id
• —0000 means that it is the first segment of that episode
• —5664 is the starting sample of this segment. Remember all episodes are sampled at 16k Hz, so the total number of samples in an episode is (the duration in seconds * 16,000).
• —81376 is the ending (exclusive) sample of this segment.

Metadata

Metadata for each episode is stored in the metadata.jsonl file, where each line stores the metadata for one episode: Here's the metadata for one of the episodes (split into multiple lines for clarity):

{
  "title": "SBS 中文新聞 （7月5日）",
  "date": "05/07/2023",
  "view_more_link": "https://www.sbs.com.au/language/chinese/zh-hant/podcast-episode/chinese-news-5-7-2023/tl6s68rdk",
  "download_link": "https://sbs-podcast.streamguys1.com/sbs-cantonese/20230705105920-cantonese-0288b7c2-cb6d-4e0e-aec2-2680dd8738e0.mp3?awCollectionId=sbs-cantonese&awGenre=News&awEpisodeId=20230705105920-cantonese-0288b7c2-cb6d-4e0e-aec2-2680dd8738e0"
}

where

• —title is the title of the episode
• —date is the date when the episode is published
• —view_more_link is a link to the associated article/description for this episode. Many news episodes have extremely detailed manuscripts written in Traditional Chinese while others have briefer summaries or key points available.
• —download_link is the link to download the audio for this episode. It is usually hosted on streamguys but some earlier episodes are stored SBS's own server at https://images.sbs.com.au.

The id of each episode appears at the end of its view_more_link. It appears to be a precomputed hash that is unique to each episode.

id = view_more_link.split("/")[-1]