Datasetpublic

Tim2190/kazakh-codeswitch-asr

Kazakh Code-Switching ASR Benchmark A benchmark for evaluating ASR systems on natural Kazakh speech that code-switches with Russian — the everyday Kazakh–Russian mixing found in stand-up, interviews and vlogs, not scripted read speech. This is, to our knowledge, the first speech/ASR resource targeting the Kazakh–Russian code-switching pair (existing Kazakh–Russian NLP resources are text-only). Code, scoring harness and full analysis:… See the full description on the dataset page: https://huggingface.co/datasets/Tim2190/kazakh-codeswitch-asr.

sourceHugging Facecc-by-4.0updated 2mo agoView on Hugging Face

0likes96downloads

Dataset Card

Kazakh Code-Switching ASR Benchmark

A benchmark for evaluating ASR systems on natural Kazakh speech that code-switches with Russian — the everyday Kazakh–Russian mixing found in stand-up, interviews and vlogs, not scripted read speech. This is, to our knowledge, the first speech/ASR resource targeting the Kazakh–Russian code-switching pair (existing Kazakh–Russian NLP resources are text-only).

Code, scoring harness and full analysis: https://github.com/Tim2190/Kaz-ASR-codeswitch-benchmark

Dataset structure

31 clips (~3.7 min total, clips ≤ 20 s), sourced from CC BY YouTube material. Each clip carries two independent reference transcripts and four binary phenomenon flags:

Field	Description
`file_name`	path to the WAV clip
`audio_id`	clip identifier
`transcript_verbatim`	literal transcript, exactly as spoken (reductions kept)
`transcript_normalized_written`	dictionary word forms
`has_contraction`	fast-speech reductions present
`has_dialect_slang`	dialectal/slang lexis present
`has_barbarisms`	a Russian insertion where a native Kazakh word exists
`has_propers`	proper nouns present
`audio_source_link`	source channel/title (CC BY attribution)

Audio is 48 kHz stereo 16-bit WAV. The two-layer design separates genuine recognition errors from a system's tendency to auto-normalize morphology. Full annotation rules are in annotation_methodology.md in the GitHub repo.

Loading

python

from datasets import load_dataset
ds = load_dataset("Tim2190/kazakh-codeswitch-asr", split="train")
print(ds[0]["audio"], ds[0]["transcript_verbatim"])

Benchmark results (5 systems, WER / CER, lower is better)

System	Type	WER	CER
Fine-tuned Kazakh Whisper	fine-tuned	11.9%	3.8%
Yandex SpeechKit	commercial STT	17.1%	6.5%
Gemini 2.5 Flash	commercial LLM	29.5%	12.5%
Whisper large-v3	base, zero-shot	42.5%	13.0%
Google Cloud STT	commercial STT	64.9%	38.1%

Key finding: commercial services span the entire quality range (Yandex 2nd, Google last), and Russian insertions are the dominant error driver — errors concentrate on the switched-in Russian words. See the GitHub repo for the per-phenomenon breakdown and word-level analysis.

License & attribution

Source audio is derived from Creative Commons CC BY YouTube material; the originating channel/title for every clip is in the audio_source_link field. Annotations and code are MIT-licensed. Created by Timur Seidalin.