failed09/bashkir-wikipedia-monolingual

Bashkir Wikipedia Monolingual Corpus

Cleaned sentence-level Bashkir text from Wikipedia for pretraining, tokenizer training and linguistic research.

Overview

Sentence-level text extracted from the Bashkir Wikipedia dump (bawiki-20260801), cleaned and filtered with automated language identification. The cleaned configuration is the recommended default for language modelling, tokenization and linguistic research; precleaned is an earlier, lighter extraction kept for alternative preprocessing and comparison.

Contents

Files and Configurations

Counts and artifact hashes are kept in `META.json` as the source of truth.

Schema

cleaned/data.parquet:

precleaned/data.parquet contains title and text.

Examples

Method

bawiki-20260801 dump → wikitext stripping → Unicode NFC normalization
    ├─ language gate + sentence filters → cleaned/data.parquet    (recommended)
    └─ lighter baseline filtering      → precleaned/data.parquet

• —Extraction: sentences extracted from the official Wikimedia dump bawiki-20260801.
• —Wikitext stripping: MediaWiki markup, templates, infoboxes, tables, external links and reference tags removed.
• —Normalization: Unicode NFC normalization, standardized quotes and balanced punctuation; all nine Bashkir-specific letters (Ә Ғ Ҙ Ҡ Ң Ө Ҫ Ү Һ) preserved.
• —Language filtering: non-Bashkir admixture removed with Bashkir LID (ba_prob >= 0.50); see the companion Bashkir LID.
• —Sentence boundaries: incomplete fragments, orphaned list entries and snippets under 5 words are excluded from cleaned only.

cleaned applies the full language gate and sentence filters. precleaned stops at the earlier, lighter baseline stage and is intentionally noisier.

Quality and Use

Non-Bashkir material was partially removed with automatic filtering. At this scale, encyclopedic texts cover global geography, history and sciences, so some borrowed vocabulary, foreign names, technical terms or segmentation artifacts may remain. Review samples before using the dataset in production or for evaluation benchmarks.

Limitations

• —Automated language filtering can miss short, ambiguous or code-mixed sentences.
• —train is a distribution convention; no leakage-free evaluation split is provided.
• —precleaned is intentionally noisier than cleaned and is meant for custom filtering experiments.

Related Resources

• —Bashkir-Russian Wikipedia Parallel Corpus — sentence-aligned Bashkir-Russian pairs from the same dump, for translation and cross-lingual tasks.

Usage

pip install datasets

from datasets import load_dataset

dataset = load_dataset("failed09/bashkir-wikipedia-monolingual", "cleaned")
df = dataset["train"].to_pandas()
print(df.head())

Or directly with Pandas:

import pandas as pd

df = pd.read_parquet("cleaned/data.parquet")

License

Distributed under the CC BY-SA 4.0 license. The release contains text extracted from the Bashkir Wikipedia; upstream source licenses and attribution requirements apply. Preserve Wikipedia and Wikimedia attribution when redistributing or publishing derivative datasets.

Citation

@dataset{failed09_bashkir_wikipedia_monolingual_2026,
  title = {Bashkir Wikipedia Monolingual Corpus},
  author = {failed09},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/datasets/failed09/bashkir-wikipedia-monolingual},
  note = {Cleaned sentence-level Bashkir Wikipedia corpus}
}

Open Bashkir Data and Sources 🐝

This release is part of an open-source effort to support the development, preservation and practical use of the Bashkir language. Other related models, datasets and tools are available on the author's Hugging Face profile.

The author does not claim ownership or authorship of the source texts or other materials used to derive this release; rights and licensing remain with the original authors, publishers and dataset providers. Source texts are not redistributed in this repository, so users should follow the licenses and attribution requirements of the relevant upstream resources.