NCSpeech/YO-CPT-kk

YO-CPT-kk

YouTube-Oriented dataset for Continual Pre-Training (Kazakh). A heavily quality-filtered corpus of Kazakh speech mined from YouTube and processed into clean, single-speaker, TTS-grade utterances. Every utterance ships with an ensemble-verified transcription, a punctuated/denormalized and stress-marked text variant, word-level forced alignment, within- and cross-video speaker identities, an audio-quality (MOS) score, and a speaker persona built from the voice and, where available, the speaker's on-screen face.

Kazakh is a low-resource language, so we collected the source audio ourselves by searching and crawling Kazakh-language YouTube — 143,735 videos / 47,141 hours in total. This release takes 8,053 whole videos from that pool and processes them into 600 hours / 156,903 utterances (per-stage numbers in the pipeline below).

• —156,903 utterances · 600 hours, single-speaker chunks, 24 kHz mono
• —Primary use — continual pre-training (CPT) of TTS models: TTS-grade audio (denoised, loudness-normalized, edge-clean, force-aligned) with rich text and speaker conditioning.
• —Also suitable for ASR (ensemble-agreed transcripts), Speaker Verification (within/cross-video speaker ids), and Turn-Detection (single-speaker segmentation + word/pause timings).

Audio is embedded (HF Audio feature). Shards hold ~4,000 utterances each (kk-XXX.parquet).

Pipeline

Three stages take raw YouTube audio to the finished corpus:

Kazakh YouTube  —  targeted search + crawl   ·   143,735 videos · 47,141 hours downloaded
   │
   │   this release: 8,053 whole videos  ·  2,678 hours of source video
   ▼
STAGE 1 · ASR mining
   VAD chunking (Silero) → speaker consistency (VoxBlink2) → quality MOS ≥ 3.0 (DistillMOS)
   → ASR ensemble (Whisper-large-v3-turbo-ru-kk · wav2vec2-BERT CTC · FastConformer-Hybrid ru-kk)
   → Kazakh-language filter → WER cross-check + ROVER
   │
   ▼  clean single-speaker utterances + ensemble-verified text   ·   784 h · 210K utts
STAGE 2 · TTS processing
   anti-spoofing (Spectra-0) → enhancement (ClearVoice MossFormer2) → loudness norm + clip guard
   → edge check + recrop (re-ASR reconcile) → forced alignment (wav2vec2-BERT, + [pause])
   │
   ▼  TTS-grade audio + word/pause alignment   ·   600 h · 156,903 utts
STAGE 3 · Metadata enrichment
   stress (RUAccent + rule-based Kazakh accentuator) · denorm + punctuation (GPT-4.1-mini)
   speaker persona:  face (TalkNet → LVFace) + face description (Qwen3.5-Flash) + voice (librosa + ECAPA gender)  →  spk_desc
   │
   ▼
YO-CPT-kk (this release)  ·  156,903 utterances  ·  600 hours

Stage 1 — ASR mining

Turns full-length YouTube audio into clean, transcribed, single-speaker utterances.

• —VAD chunking — Silero VAD segments each full-length video into utterance-length pieces; the maximum segment length is randomized per file so cut points don't land on a fixed grid, and over-long / over-short segments are dropped.
• —Speaker consistency — VoxBlink2 (ResNet34) speaker embeddings are computed over sliding windows, and a chunk is discarded entirely if it contains speech from more than one speaker; the same check also drops non-speech chunks.
• —Audio quality — DistillMOS no-reference MOS estimation; chunks below MOS 3.0 are discarded.
• —ASR ensemble — three deliberately different systems, each adapted for Kazakh in-house, transcribe every chunk: a fine-tuned Whisper-large-v3-turbo (attention encoder–decoder), a custom wav2vec2-BERT CTC model, and a bilingual ru-kk FastConformer-Hybrid (Transducer+CTC, NeMo). Architectural diversity makes correlated errors unlikely, so agreement is a strong signal of correctness.
• —Kazakh-language filter — Kazakh YouTube is heavily bilingual, and many videos are partly or wholly in Russian: fully Russian chunks were filtered out of the dataset. Natural Kazakh–Russian code-switching within Kazakh speech is preserved.
• —Cross-validation + ROVER — pairwise WER is computed between the three hypotheses; chunks without close agreement between at least two systems are dropped. On tight agreement the transcription is trusted as-is; on moderate disagreement the three hypotheses are aggregated by ROVER word-level voting.

Stage 2 — TTS processing

Turns transcribed utterances into TTS-grade audio with word-level alignment.

• —Anti-spoofing — runs first, on the audio before enhancement, which could otherwise mask synthesis artifacts. A Spectra-0 spoof detector scores each chunk, dropping likely-synthetic speech (see below).
• —Speech enhancement — ClearVoice MossFormer2_SE_48K denoises and dereverberates audio while preserving the target speaker.
• —Loudness normalization + clipping guard — loudness is normalized to a consistent target with peak limiting; chunks with too high a clipped-sample ratio are rejected outright.
• —VAD edge check + recrop — a second VAD pass trims each clip to the nearest internal silence boundary (or discards it if that's impossible); recropped audio is re-transcribed and reconciled with the original transcription to ensure trimming has not altered the content.
• —Forced alignment — a custom wav2vec2-BERT CTC model produces word-level start/end timings, inserting explicit [pause] tags at internal silences. This provides the text_alignment required by TTS models for duration modeling.

Stage 3 — Metadata enrichment

• —Stress — Kazakh word stress typically falls on the final syllable; for precise placement (suffixes, particles and other exceptions) we built a rule-based accentuator, released with the dataset (accent_worker.py). Embedded Russian words are stressed with RUAccent. Marks are carried into text_denorm and text_alignment.
• —Denormalization + punctuation — OpenAI gpt-4.1-mini restores casing/punctuation and denormalizes the raw ASR text → text_denorm.
• —Speaker persona — a static per-speaker profile from face + voice. The identity design is based on voice locally, faces globally: a voice reliably separates speakers within one video, while a face is what re-identifies the same person across different videos.
• —Voice → local id — VoxBlink2 embeddings cluster the speakers within each video → local_spk_id.
• —Face → global id — TalkNet selects the active-speaker (actually-talking) face, LVFace embeds it, and these are clustered across videos into the speaker's cross-video global_spk_id (null for audio-only speakers).
• —Audio signal — gender via an ECAPA-TDNN voice classifier; librosa features: median F0 → pitch, spectral centroid → brightness, spectral flatness → breathiness, words/sec → rate.
• —Visual description — head-and-shoulders crops described by Qwen3.5-Flash (gender, age, nationality, appearance, style).
• —Fuse — face demographics (if real and live) + voice traits → structured fields plus a natural-language spk_desc.

Implementation details

Speech enhancement. ClearVoice MossFormer2 was chosen over two alternatives on a benchmark subset sampled from the corpus (100 clips). Automatic metrics: WER between Whisper transcripts of the original vs the enhanced clip, and speaker similarity via VoxBlink2 embeddings. The last two columns report manual listening evaluations — how much noise the model removes and how natural the enhanced audio sounds:

Anti-spoofing. Spectra-0 scores each chunk (higher = more likely genuine). The distribution has two obvious centers — a bonafide mode near +5 and a low-scoring mode near −7.5:

<img src="figures/figspoofkk.png" alt="Anti-spoofing — bonafide score distribution (random 50k KZ sample)" width="560">

The drop threshold (−2.5) was chosen by manually auditioning low-scoring chunks; 7.2% of chunks fall below it and are dropped (an additional per-video mean check removes videos dominated by synthetic speech).

Face validation. Matching a face to a voice is error-prone: a frame may contain several faces, and channels often show posters or static photos of someone other than the speaker. A face is attached to a speaker only after passing the full validation funnel (TalkNet lip-sync active-speaker detection → face-embedding quality checks → realness check dropping cartoons, posters and static photos). In this release:

• —11,839 — speakers total;
• —4,436 (37%) — with a confidently talking on-screen face: TalkNet-ASD active-speaker detection accepts a face only if its lip motion is synchronized with the audio;
• —4,258 (36%) — after the realness check (drops cartoons, posters, static photos).

Only these speakers carry the face-derived fields (image_desc, global_spk_id); everyone else stays audio-only.

Speakers. Two levels of annotation.

`local_spk_id` — unique within a single video: 11,839 within-video speakers across 8,053 videos, obtained by agglomerative clustering (AHC, average linkage) of VoxBlink2 voice embeddings within each video. Because utterances containing more than one speaker were filtered out upstream (Stage 1), within-video speaker labelling is reliable.

`global_spk_id` — the same person linked across videos through the face modality only (voice-only cross-video matching would introduce a substantial error rate): the 4,258 face-validated speakers collapse into 2,947 global identities, of which 211 appear in at least 3 videos.

Data format

from datasets import load_dataset
ds = load_dataset("NCSpeech/YO-CPT-kk", split="train", streaming=True)

A sample record:

{
  "utt_id": "JbLec_5Bf0Y_0_68",
  "text": "сондықтан келем десеңіздер міндетті түрде төменде жазыңыздар і төменде бүкіл ақпарат бар",
  "text_denorm": "Сондықт+ан кел+ем десеңізд+ер, міндетт+і түрд+е төменд+е жазыңызд+ар. +І төменд+е бүк+іл ақпар+ат б+ар.",
  "text_alignment": "сондықт[+]ан(0.10,0.74) кел[+]ем(0.76,1.06) десеңізд[+]ер(1.08,1.62) міндетт[+]і(1.64,2.02) түрд[+]е(2.04,2.38) [pause0.3](2.38,2.64) төменд[+]е(2.64,3.18) [pause0.7](3.18,3.86) жазыңызд[+]ар(3.86,4.58) [pause0.4](4.58,5.00) [+]і(5.00,5.22) төменд[+]е(5.24,5.62) бүк[+]іл(5.64,5.88) ақпар[+]ат(5.90,6.26) б[+]ар(6.28,6.56)",
  "duration": 6.67,
  "local_spk_id": "JbLec_5Bf0Y__spk0",
  "global_spk_id": "spk_93",
  "lang": "kk",
  "spk_desc": {
    "audio_only": false,
    "audio_desc": {
      "gender": "female",
      "f0_hz": 176.4,
      "pitch": "low",
      "rate": "moderate",
      "brightness": "moderate",
      "breathiness": "slightly-breathy",
      "text": "female voice: low pitch, moderate, slightly-breathy, moderate pace; speaks Kazakh"
    },
    "image_desc": {
      "gender": "female",
      "age": "30-45",
      "nationality": "Other",
      "appearance": "Medium skin tone, dark wavy hair, dark eyes, wearing large blue over-ear headphones.",
      "style": "Casual green patterned shirt, informal grooming.",
      "impression": "Casual home video participant",
      "text": "Other female (30-45). Casual green patterned shirt, informal grooming."
    }
  },
  "mos_score": 3.104
}

Each record has the following fields:

spk_desc format

Audio-only speakers carry "audio_only": true and "image_desc": null.

Bias, Risks, and Limitations

• —Automatic annotation. Every label is model-generated and not human-verified — human involvement was limited to tuning the pipeline's hyperparameters (e.g. filter thresholds) — so minor errors are expected: occasional inaccuracies in transcriptions, in text processing (punctuation, stress marks, alignment timings), and in speaker metadata. A segment kept as single-speaker may still contain brief intrusions of a second voice where one speaker clearly predominates.
• —Visual persona is fragment-scoped. The face-derived description is inferred from a few frames of a single video clip and may not represent the person in general.
• —Persona attributes are apparent, not factual. All demographic fields in spk_desc are automatic estimates of how the speaker sounds or looks; they can be wrong and must not be treated as facts about the person.
• —Source representativeness. The corpus mirrors the distribution of Kazakh-language YouTube content: some speakers, accents, domains, and recording conditions are over- or under-represented, and recurring presenters contribute disproportionately many utterances.
• —Anonymized identifiers. Video identifiers inside utt_id / local_spk_id are anonymized and do not correspond to real YouTube video ids.

License

YO-CPT-kk is a derived dataset: beyond attribution, the authors impose no restrictions of their own — users are responsible for complying with the upstream licenses below.

Contribution of the authors. The annotations and the compilation of the corpus are released under CC BY 4.0.

Source recordings. The source audio was collected by the authors from publicly available YouTube videos. The copyright of the recordings remains with their original owners; this dataset conveys no rights to the underlying recordings beyond the authors' own contribution (see Contact for removal requests).

Upstream components. The released audio and metadata are produced by the following components:

Contact

For questions about the dataset, or to request removal of your material (as a rights holder or as a person appearing in the recordings), contact us at aleksei.gusev@ncspeech.org.

Citation

If you use this dataset, please cite it as follows.

@misc{yocptkk2026,
  title        = {YO-CPT-kk: A YouTube-Oriented Kazakh Speech Corpus for Continual Pre-Training},
  author       = {{NCSpeech team}},
  year         = {2026},
  howpublished = {Hugging Face Hub},
  url          = {https://huggingface.co/datasets/NCSpeech/YO-CPT-kk}
}