CoolFace
Datasetpublic

ansarzeinulla/Nogai-Unified-Corpus-v1

Nogai Unified Corpus (NUC) v1 Monolingual text in Nogai (Kipchak Turkic, Cyrillic script) for continued pre-training of language models. It was built for NogaiLLM. Size Rows (sentences / short paragraphs) 163,531: 155,354 train, 8,177 validation (95 / 5) Words 2.35 M Characters 18.1 M UTF-8 text 33.4 MB (files: 35.6 MB) Tokens (Qwen2.5 tokenizer) 9.8 M Format JSONL, one {"text": ...} per row Sources Newspapers: digitised… See the full description on the dataset page: https://huggingface.co/datasets/ansarzeinulla/Nogai-Unified-Corpus-v1.

sourceHugging Facecc-by-nc-4.0updated 4d agoView on Hugging Face
1likes71downloads
Dataset Card

Nogai Unified Corpus (NUC) v1

Monolingual text in Nogai (Kipchak Turkic, Cyrillic script) for continued pre-training of language models. It was built for NogaiLLM.

Size

Rows (sentences / short paragraphs)163,531: 155,354 train, 8,177 validation (95 / 5)
Words2.35 M
Characters18.1 M
UTF-8 text33.4 MB (files: 35.6 MB)
Tokens (Qwen2.5 tokenizer)9.8 M
FormatJSONL, one {"text": ...} per row

Sources

  • —Newspapers: digitised issues of Шоьл тавысы / Голос степи (Dagestan, 2015–2026) and Ногай давысы (Karachay-Cherkessia, 2024–2026), extracted from PDF.
  • —Bible translations by the Institute for Bible Translation (IBT).
  • —Wikimedia Incubator (Wp/nog).

Processing

  1. 1.OCR and encoding repair.
  2. 2.Removal of Russian text by Russian stop-word frequency.
  3. 3.Sentence/paragraph segmentation, then shuffling and a 95/5 split.

Checks (September 2026)

  • —No exact duplicate rows. No validation row appears verbatim in train; 15 (0.2%) match after ignoring case and punctuation.
  • —Residual Russian: about 0.4% of rows still look Russian (≥20% Russian function words), e.g. official notices quoted in the newspapers. The filter removes most Russian, not all.
  • —Overlap with the SFT data: the Bible text here overlaps the Russian–Nogai SFT pairs. 13% of the Nogai SFT sentences occur verbatim in this corpus. Held-out translation tests should exclude them. The clean SFT v2 split does.
  • —Typographic Density of this human text (Nogai digraphs аь/оь/уь/нъ per 100 words): 22.7 (validation), 22.9 (train). Use it as the reference value when judging model output.

Usage

python
from datasets import load_dataset

ds = load_dataset("ansarzeinulla/Nogai-Unified-Corpus-v1")
print(ds["train"][0]["text"])

License and permissions

The dataset card and processing are CC BY-NC 4.0. The underlying texts belong to their publishers (the two newspapers and IBT). Use for non-commercial research only, and contact the publishers for anything else.

Citation

bibtex
@dataset{nogai_unified_corpus_2026,
  author       = {Zeinulla, Ansar},
  title        = {Nogai-Unified-Corpus-v1},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/ansarzeinulla/Nogai-Unified-Corpus-v1}}
}