failed09/bashkir-frequency-index

Bashkir Frequency Index v11.5

Word-frequency index for Bashkir, computed over a large monolingual Bashkir-language dataset, for NLP, spellchecking and lexical research.

Overview

Word-frequency index for the Bashkir language computed over a large monolingual Bashkir-language dataset. Non-Bashkir admixture, borrowed vocabulary and scanning artifacts were reduced with automated language filtering. The public configuration (count ≥ 3) is the recommended default; full keeps all retained forms for comprehensive lexical and morphological study.

Contents

Files and Configurations

Corpus statistics, cutoffs and artifact hashes are the source of truth in `META.json`.

Schema

Examples

Method

monolingual Bashkir text → lowercase Cyrillic tokenization → language filtering
    → counting → public (count ≥ 3) / full → frequency index

• —Tokenization: lowercase regex over Cyrillic, including all nine Bashkir-specific letters (Ә Ғ Ҙ Ҡ Ң Ө Ҫ Ү Һ).
• —Counting: absolute count and sentence-level doc_freq over the cleaned corpus.
• —Filtering: automated function-word language filtering removes foreign-language leaks.
• —Form constraints: isolated single-character forms removed (except ә); concatenated strings over 24 characters removed.
• —Configurations: public keeps forms with count ≥ 3; full keeps all retained forms.

Corpus size and token totals are recorded in `META.json`.

Quality and Use

Non-Bashkir words and other language admixture were partially removed with automatic filtering. At this scale, some words from other languages, borrowed forms, names, technical terms, regional vocabulary, OCR artifacts or noisy entries may still remain; this is normal for a large web-derived corpus. Always validate words and frequency-based decisions before using the index in production, linguistic analysis, spellchecking or a user-facing application.

Limitations

• —Presence in the index is not proof that a form is standard Bashkir.
• —Automated filtering is imperfect and can miss short or code-mixed forms.
• —Counts reflect the underlying corpus composition, not usage norms.

Related Resources

• —Bashkir Word N-gram Index — exact within-sentence unigram, bigram and trigram counts; use together with this index for phrase-level statistics.

Usage

pip install datasets

from datasets import load_dataset

dataset = load_dataset("failed09/bashkir-frequency-index", "public")
df = dataset["train"].to_pandas()
print(df.head(10))

Or directly with Pandas:

import pandas as pd

df = pd.read_parquet("frequency_public.parquet")

License

Distributed under the CC BY 4.0 license. The release contains derived statistics, not the source texts. Upstream source licenses and attribution requirements still apply to the underlying materials.

Citation

@dataset{failed09_bashkir_frequency_index_2026,
  title = {Bashkir Frequency Index v11.5},
  author = {failed09},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/datasets/failed09/bashkir-frequency-index},
  note = {Open-source Bashkir word-frequency index for corpus processing and linguistic research}
}

Open Bashkir Data and Sources 🐝

This release is part of an open-source effort to support the development, preservation and practical use of the Bashkir language. Other related models, datasets and tools are available on the author's Hugging Face profile.

The author does not claim ownership or authorship of the source texts or other materials used to derive this release; rights and licensing remain with the original authors, publishers and dataset providers. Source texts are not redistributed in this repository, so users should follow the licenses and attribution requirements of the relevant upstream resources.