alibiserikbay/kazakh-russian-mixed-stt

Kazakh & Kazakh-Russian Mixed STT — Speech Recognition for Kazakh and Russian

Kazakh speech recognition (ASR / STT) models for Kazakh (қазақ тілі), Russian (русский язык), and mixed Kazakh-Russian code-switching speech. Built on wav2vec2 + CTC and shipped as TorchScript, together with matching KenLM language models, a punctuation restorer, and a voice-activity detector.

Kazakh is a low-resource language for most open speech-to-text systems, and closing that gap is why I trained these. The monolingual models are small — 94M parameters — and run faster than real time on a laptop CPU with no GPU. The mixed model transcribes Kazakh-Russian code-switching in a single pass without a separate language-ID step, which is what call-center and telephony audio needs when speakers move between Kazakh and Russian mid-sentence.

Released free to use, including commercially, under Apache-2.0. If you try them, I'd be glad to hear how they do on your data.

Keywords: Kazakh ASR · Kazakh speech to text · Kazakh STT · қазақша сөйлеуді тану · Russian ASR · распознавание речи · Kazakh-Russian bilingual ASR · code-switching speech recognition · wav2vec2 Kazakh · CTC · KenLM · offline on-premise STT · CPU inference

• —Model type: wav2vec2 encoder + CTC head (TorchScript, traced)
• —Languages: Kazakh (kk), Russian (ru), mixed kk/ru
• —Parameters: 94.4M (monolingual) / 188.8M (mixed)
• —Audio: 16 kHz mono, raw waveform
• —License: Apache-2.0 — free for commercial use

Available models

Supporting components: KenLM language models (lm/), punctuation restoration (punctuation/), voice-activity detection (vad/).

config.json at the repository root is a manifest describing every component — paths, vocab sizes, parameter counts, decoder settings, and evaluation results — in machine-readable form. It is not a transformers AutoConfig: these models are TorchScript archives loaded with torch.jit.load(), not from_pretrained() checkpoints.

Quick start

Don't `git clone` this repo. It is 9.27 GB, most of which is language models you may not need. Download only the files you want — one acoustic model is 360 MB.

pip install torch huggingface_hub soundfile

from huggingface_hub import hf_hub_download
import torch, re

REPO = "alibiserikbay/kazakh-russian-mixed-stt"
LANG = "kk"          # "kk" | "ru" | "rukk"

model_path  = hf_hub_download(REPO, f"asr/{LANG}/model.pt")
tokens_path = hf_hub_download(REPO, f"asr/{LANG}/tokens.lst")

model = torch.jit.load(model_path, map_location="cpu").eval()

tokens = {}
for line in open(tokens_path, encoding="utf-8"):
    if line.strip():
        sym, idx = line.rstrip("\n").split("\t")
        tokens[int(idx)] = sym
blank = max(tokens) + 1          # CTC blank is the last index

def transcribe(wav):             # wav: float32 numpy array, 16 kHz mono
    with torch.no_grad():
        logits = model(torch.from_numpy(wav).unsqueeze(0))[0]
    ids, out, prev = logits[0].argmax(-1).tolist(), [], None
    for i in ids:                # collapse repeats, drop blanks
        if i != prev and i != blank:
            out.append(tokens.get(i, ""))
        prev = i
    text = "".join(out).replace("|", " ").replace("_", " ")
    return re.sub(r"\s+", " ", text).strip()

import soundfile as sf
wav, sr = sf.read("audio.wav", dtype="float32")
assert sr == 16000, "resample to 16 kHz first"
if wav.ndim > 1:
    wav = wav.mean(1)            # to mono
print(transcribe(wav))

That is the complete greedy pipeline. For best accuracy add the KenLM decoder — see Recommended: KenLM beam-search decoding, which costs ~4 WER points less error but a larger download.

What to download

Grab a whole subtree with allow_patterns:

from huggingface_hub import snapshot_download
snapshot_download(REPO, allow_patterns=["asr/kk/*", "lm/kk/*", "config.json"])

Performance

Evaluated on the FLEURS test sets. All numbers below were measured directly on this release — none are carried over from another report. Text normalization: lowercase, punctuation stripped, ё→е.

Kazakh — FLEURS kk_kz (n = 856)

KenLM beam-search decoding is worth −4.10 WER points (−24% relative) over greedy on the full test set. It is strongly recommended — the greedy numbers are a floor, not the intended operating point.

Note that the mixed model slightly outperforms the dedicated Kazakh model on Kazakh (16.01% vs 16.88% greedy WER). Its second encoder tower appears to help rather than dilute.

Russian — FLEURS ru_ru (n = 775)

On pure Russian the mixed model gives up ~1.5 WER points to the dedicated Russian model — the cost of covering both languages in one pass.

Comparison

Read this table carefully. The Whisper-turbo figures are self-reported from that model's card, not measured here, and its text normalization may differ — so this is an indicative comparison, not a controlled head-to-head.

What the numbers do support: at 8.5× fewer parameters this model reaches a marginally lower character error rate and lands within 1 WER point. Two structural caveats matter when interpreting the remaining WER gap:

1. 1.Number formatting. This model transcribes numbers as spoken words (бір мың тоғыз жүз қырықыншы), while FLEURS references use digits (1940). Each number costs several word errors despite being a correct transcription. Whisper, trained on written text, emits digits natively and avoids the penalty. This inflates our WER while barely touching CER — which is exactly the pattern observed. Inverse text normalization would recover part of it.
2. 2.Architecture. Whisper is an encoder-decoder with an implicit internal language model. This is a CTC model that externalizes that role to KenLM, so comparing CTC-greedy against Whisper compares an incomplete system to a complete one.

Inference speed

Measured on an Apple M1 Pro (6 performance + 2 efficiency cores, 16 GB), PyTorch 2.10, FP32, batch 1. Median of 5 runs after 2 warm-ups. ×RT = times faster than real time.

(10 s audio. Full per-duration data in `benchmarks/`.)

Notes that matter in production:

• —Threading scales sub-linearly — 8 threads buys ~2×, not 8×. On a multi-core server, run several single-threaded workers in parallel rather than giving one worker every core.
• —Longer audio is less efficient per second (attention is quadratic). Chunk to ~10–20 s using the bundled VAD.
• —These are acoustic forward-pass figures only. KenLM beam search at beam_size: 1000 adds substantial cost and often dominates end-to-end latency (measured ~12.8×RT for the full Kazakh pipeline including decoding).
• —No CUDA numbers are published because none were measured — the benchmark machine has no NVIDIA GPU. The models were built for CUDA via Triton and should be considerably faster there, but a number we did not measure would be a guess.
• —Apple MPS fails above ~20 s of audio (Output channels > 65536). Chunk, or use CPU/CUDA. Verified working at 20 s, failing at 25 s.

Architecture

All acoustic models are wav2vec2 encoders with a CTC head, exported with torch.jit.trace from transformers 4.37. Input is a raw 16 kHz mono waveform (float32, [batch, samples]) — no filterbank front-end. Output is per-frame CTC logits at 20 ms/frame (49 frames per second).

Kazakh and Russian — wav2vec2-base

Mixed kk+ru — dual-encoder ensemble

The mixed model is not a single larger network. Two complete, independent wav2vec2-base towers process the same waveform; their 768-dim outputs are concatenated to 1536 and a single shared CTC head projects to 47 classes.

                    ┌──────────────────────┐
 waveform 16 kHz ──►│ model1 (w2v2-base)   ├──► 768 ┐
        │           └──────────────────────┘        ├─ concat ─► 1536 ─► Linear ─► 47
        │           ┌──────────────────────┐        │
        └──────────►│ model2 (w2v2-base)   ├──► 768 ┘
                    └──────────────────────┘

This costs ~2× the compute of a single base model but lets the towers specialize, which is the mechanism for handling Russian/Kazakh code-switching within one utterance.

Token inventories

CTC classes = tokens + 1 blank. The blank is the last index.

• —Kazakh / mixed — 47 classes. Russian Cyrillic plus the 9 Kazakh letters ә ғ қ ң ө ұ ү һ і, plus -, _, | (word separator), [UNK].
• —Russian — 38 classes. 33 Russian Cyrillic letters, -, _, |, [UNK].

The Kazakh and mixed models share the same 47-symbol inventory, so either can emit Russian text; they differ in training emphasis and structure, not alphabet.

Usage

Basic transcription (greedy CTC)

See Quick start above for the complete copy-paste example. Greedy decoding is a floor, not the intended operating point — it costs ~4 WER points versus the LM decoder below.

Recommended: KenLM beam-search decoding

pip install torchaudio flashlight-text kenlm

from huggingface_hub import snapshot_download
# pulls asr/kk (360 MB) + lm/kk (2.1 GB LM + 202 MB lexicon)
local = snapshot_download(REPO, allow_patterns=["asr/kk/*", "lm/kk/*"])

The lm/*/lm.bin files are already-compiled KenLM binaries (probing hash tables), not ARPA text. Do not run build_binary on them — load them directly.

import torch
from torchaudio.models.decoder import ctc_decoder   # needs flashlight-text

tok = {}
for line in open(f"{local}/asr/kk/tokens.lst", encoding="utf-8"):
    if line.strip():
        s, i = line.rstrip("\n").split("\t"); tok[int(i)] = s
token_list = [tok[i] for i in range(max(tok) + 1)] + ["<blank>"]

decoder = ctc_decoder(
    lexicon=f"{local}/lm/kk/words.lst",   # already in flashlight format
    tokens=token_list,
    lm=f"{local}/lm/kk/lm.bin",
    nbest=1,
    beam_size=1000, beam_size_token=1000, beam_threshold=15,
    lm_weight=1.2, word_score=1.0, sil_score=-0.2,
    blank_token="<blank>", sil_token="|", unk_word="[UNK]",
)

model = torch.jit.load(f"{local}/asr/kk/model.pt", map_location="cpu").eval()
with torch.no_grad():
    logits = model(torch.from_numpy(wav).unsqueeze(0))[0]
result = decoder(torch.log_softmax(logits, dim=-1).cpu())
print(" ".join(result[0][0].words))

Note blank_token="<blank>" — the default "-" would collide with the real hyphen character in these vocabularies. Trie construction over the full lexicon takes ~3 minutes and is a one-time cost per process.

Production hyperparameters (from lm/*/passport.json):

Voice activity detection

vad/vad.onnx (1.7 MB, Silero-architecture LSTM) gates the recognizer and splits long audio into the ~10–20 s chunks the acoustic models handle most efficiently. Inputs: input (waveform chunk), sr, and LSTM states h, c (both [2, batch, 64]); outputs a speech probability plus updated states. Window: 1536 samples.

import numpy as np, onnxruntime as ort
from huggingface_hub import hf_hub_download

sess = ort.InferenceSession(hf_hub_download(REPO, "vad/vad.onnx"))
h = c = np.zeros((2, 1, 64), np.float32)
for i in range(0, len(wav) - 1536, 1536):
    p, h, c = sess.run(None, {"input": wav[i:i+1536][None, :],
                              "sr": np.array(16000, dtype=np.int64), "h": h, "c": c})
    # p[0,0] > 0.5  =>  speech

Verified behaviour: speech ≈0.56 mean probability, silence ≈0.03, noise ≈0.03.

Serving with Triton

model_repository/
  asr_kk_16000/
    config.pbtxt
    1/model.pt

Config files are included at asr/*/config.pbtxt. Set the output dimension to 47 for Kazakh and mixed, 38 for Russian.

Punctuation restoration (Russian only)

punctuation/ restores , . ? over recognizer output. It uses a YouTokenToMe BPE model (bpe_16k-4.model, 16k vocab — despite the .model extension this is not SentencePiece), fragments of 150 tokens with stride 80, and a TensorFlow SavedModel.

This component is Russian-only. Its BPE vocabulary contains 43 characters — Russian Cyrillic plus a few Latin letters — and none of the Kazakh-specific letters (ә ғ қ ң ө ұ ү һ і). Kazakh text will be mapped largely to unknown tokens. It also predates the acoustic models by roughly two years. Validate before relying on it, and do not expect it to punctuate Kazakh.

Training data

Trained on a mixture of public corpora and a private corpus of my own:

ISSAI KSC2 is a public, industrial-scale open-source Kazakh corpus from the Institute of Smart Systems and Artificial Intelligence, Nazarbayev University. Listed as MIT on the dataset page and CC BY 4.0 on the IS2AI repository. Thanks to the ISSAI team for releasing it — please cite their KSC2 paper, see Citation.

YO-CPT-ru (~6,052 hours, ~1.63M utterances of Russian speech) is released by NCSpeech under CC BY 4.0 for the annotations and corpus compilation. Its audio derives from YODAS2 (CC BY 3.0), sourced from YouTube videos published under Creative Commons licences; those recordings remain the intellectual property of their original creators, who retain the right to request removal. Both licences permit commercial use and derivative works, including trained models, with attribution — which is given here and in Citation. If you build on these weights, you inherit the responsibility to comply with those upstream licences.

The Kazakh-Russian code-switching capability of the mixed model comes from a private corpus of my own, which I am not releasing and whose composition I am not disclosing.

To be straightforward about what withholding that portion costs you:

• —You cannot fully audit the private corpus's size, domain balance, speaker demographics, dialect coverage, or recording conditions.
• —Bias and coverage gaps have not been characterized. Performance may vary across accents, age groups, genders, and regional dialects of Kazakh in undocumented ways.
• —The models are marked "model_type": "telephony", so the training audio is predominantly narrowband conversational speech.

The weights are Apache-2.0 and free to use commercially — only the private portion of the training data is withheld.

Intended use

• —Transcribing call-center and telephony audio in Kazakh, Russian, or a mix of both
• —Bilingual / code-switching transcription without a separate language-ID step
• —Offline or on-premise STT where sending audio to a cloud API is unacceptable
• —CPU-only deployment at scale, where GPU budget is the binding constraint
• —A starting point for fine-tuning on a narrower domain

Out of scope

• —High-stakes automated decisions. Do not drive legal, medical, employment, immigration, credit, or law-enforcement outcomes from these transcripts without a human in the loop. Accuracy across speaker demographics has not been characterized.
• —Speaker identification or biometrics. These are transcription models only.
• —Covert recording or surveillance. Transcribing people without their knowledge or consent is illegal in many jurisdictions.
• —Languages other than Kazakh and Russian. The inventory is Cyrillic-only; other scripts and languages produce garbage rather than an error.
• —Non-speech audio. Gate with the bundled VAD.

Limitations

• —Domain. Tuned for narrowband conversational telephone speech. Expect degradation on wideband studio audio, far-field and meeting recordings, and heavily accented or disfluent input. The FLEURS scores above are out-of-domain read speech — in-domain telephony performance is likely better, but has not been measured here.
• —Sample rate is fixed at 16 kHz. No internal resampling; another rate silently produces bad output rather than an error.
• —No inverse text normalization. Numbers, dates, and entities come out as spoken words, not digits. Add an ITN step if you need written-form output.
• —Lowercase, unpunctuated output. Casing and punctuation need the separate punctuation model, which is Russian-only.
• —No confidence scores or word timestamps beyond raw per-frame logits, from which frame-level alignments can be derived at 20 ms resolution.
• —FP32 only. No quantized or ONNX exports. INT8 dynamic quantization on CPU is an obvious win for anyone who wants to pursue it.
• —Apple MPS caps at ~20 s of audio per call.
• —Beam-search decoding is memory-hungry — the mixed-model lexicon has 4.7M entries and its LM is 3.1 GB.

Evaluation notes

FLEURS results were produced with greedy CTC and with torchaudio's flashlight-backed ctc_decoder using the production hyperparameters above. References and hypotheses were normalized identically: lowercased, punctuation stripped, ё→е, and the non-lexical _ marker (which these models emit at silence boundaries) removed. That last step matters — if you benchmark these yourself, leaving _ in place inflates WER by roughly 2×, because Python's \w treats underscore as a word character.

Raw latency measurements are in benchmarks/cpu.json and benchmarks/mps.json.

Corrections and additional benchmarks — especially CUDA timings and in-domain WER — are welcome via the Community tab.

Known issues

Kazakh Triton config output width. The original deployment's config.pbtxt for the Kazakh model declared dims: [-1, -1, 38], but its CTC head is Linear(768 → 47) and it emits 47 classes, consistent with its 47-entry tokens.lst. The config in this repository is corrected to 47. If you are running these weights from an older config, check this field.

License

Apache License 2.0. You may use, modify, redistribute, and build commercial products on these models and their derivatives, provided you retain the license and attribution. Developed as an independent personal project; I hold the rights to the released weights.

Citation

@misc{kazakh_russian_mixed_stt,
  title  = {Kazakh and Kazakh-Russian Mixed STT},
  author = {alibiserikbay},
  year   = {2026},
  url    = {https://huggingface.co/alibiserikbay/kazakh-russian-mixed-stt},
  note   = {wav2vec2 + CTC acoustic models with KenLM beam-search decoding}
}

Training corpora

Please also credit the public corpora used in training.

ISSAI Kazakh Speech Corpus 2:

@inproceedings{mussakhojayeva22,
  author    = {Saida Mussakhojayeva and Yerbolat Khassanov and Huseyin Atakan Varol},
  title     = {KSC2: An Industrial-Scale Open-Source Kazakh Speech Corpus},
  booktitle = {Interspeech},
  year      = {2022}
}

YO-CPT-ru — NCSpeech, licensed CC BY 4.0 for the annotations and corpus compilation, derived from YODAS2 (CC BY 3.0) audio sourced from Creative Commons-licensed YouTube videos. Available at <https://huggingface.co/datasets/NCSpeech/YO-CPT-ru>. The dataset publishes no BibTeX entry; attribute it as:

YO-CPT-ru, NCSpeech, CC BY 4.0, derived from YODAS2 (CC BY 3.0).