failed09/bashkir-wikipedia-parallel

Bashkir-Russian Wikipedia Parallel Corpus

Sentence-level Bashkir-Russian parallel text from Wikipedia, scored and filtered for machine translation.

Overview

Sentence-level Bashkir-Russian parallel dataset extracted from the corresponding Bashkir and Russian Wikipedia dumps dated 2026-08-01. Candidate pairs are scored for semantic alignment with multilingual sentence encoders (Meta LASER3, Google LaBSE) and the in-domain Bashkir-Russian Pair Scorer. The filtered configuration is the high-precision subset recommended for machine translation; cleaned, scored and precleaned are provided for custom filtering experiments.

Contents

Files and Configurations

Counts, filter rules and score distributions are the source of truth in `META.json`.

Fields and Schema

• —ba_title, ru_title: article titles in the two Wikipedias.
• —section: section category (lead or facts).
• —ba, ru: aligned sentence pair.
• —ba_sentence_index, ru_sentence_index: sentence positions in the source articles.
• —laser_cos, laser_margin, laser_margin_xling: LASER3 cosine and margin scores (Artetxe & Schwenk, 2018).
• —labse_cos, labse_margin_xling: LaBSE cosine and cross-lingual margin (present only in filtered).
• —pair_score: semantic compatibility from the Bashkir-Russian Pair Scorer medium model, 0.0–1.0.

Examples

Method

bawiki-20260801 + ruwiki-20260801 → interlanguage article links → sentence mapping
    → markup sanitization → LASER3 / LaBSE / pair-scorer alignment scoring
    → two-voice filter → filtered/data.parquet

• —Extraction: corresponding Bashkir (bawiki-20260801) and Russian (ruwiki-20260801) articles linked through interlanguage links.
• —Sentence mapping: article-level mapping between Bashkir and Russian sentences.
• —Text sanitization: MediaWiki markup, tables, citation markers and broken punctuation stripped.
• —Multilingual scoring: pairs scored with LASER3 (bak_Cyrl.v1 + laser2), Google LaBSE and the Bashkir-Russian Pair Scorer medium.
• —Filtering: filtered keeps pairs that pass two-voice scoring (drop when laser_margin_xling < 0.876 and labse_margin_xling < 0.723) and adds pair_score.

Quality and Use

Alignment quality is estimated automatically and is not human-verified. At this scale, encyclopedic texts cover global geography, history and sciences, so some sentence pairs may be loosely aligned or contain borrowed vocabulary, foreign names or segmentation artifacts. Review samples before using the dataset in production or as an evaluation benchmark.

Limitations

• —Alignment scores are model estimates, not human judgments; thresholds were calibrated on this corpus.
• —pair_score values are not calibrated human probabilities.
• —train is a distribution convention; no leakage-free evaluation split is provided.
• —precleaned and cleaned are intentionally looser than filtered.

Related Resources

• —Bashkir Wikipedia Monolingual Corpus — cleaned monolingual sentences from the same dump, for pretraining and language modelling.
• —Bashkir-Russian Pair Scorer — standalone compatibility model used to produce the pair_score column.

Usage

pip install datasets

from datasets import load_dataset

# Recommended subset for machine translation training
filtered = load_dataset("failed09/bashkir-wikipedia-parallel", "filtered")

# Custom filtering combining pair_score and LASER margin
df = filtered["train"].to_pandas()
strict_subset = df[(df.pair_score >= 0.80) & (df.laser_margin_xling >= 0.95)]
print(f"Strict high-quality subset: {len(strict_subset):,} sentence pairs")

License

Distributed under the CC BY-SA 4.0 license. The release contains text extracted from Wikipedia; upstream source licenses and attribution requirements apply. Preserve Wikipedia and Wikimedia attribution when redistributing or publishing derivative datasets.

Citation

@dataset{failed09_bashkir_russian_wikipedia_parallel_2026,
  title = {Bashkir-Russian Wikipedia Parallel Corpus},
  author = {failed09},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/datasets/failed09/bashkir-wikipedia-parallel},
  note = {Cleaned sentence-level Bashkir-Russian Wikipedia parallel corpus}
}

Open Bashkir Data and Sources 🐝

This release is part of an open-source effort to support the development, preservation and practical use of the Bashkir language. Other related models, datasets and tools are available on the author's Hugging Face profile.

The author does not claim ownership or authorship of the source texts or other materials used to derive this release; rights and licensing remain with the original authors, publishers and dataset providers. Source texts are not redistributed in this repository, so users should follow the licenses and attribution requirements of the relevant upstream resources.