failed09/bashkir-diacritics-restorer

Bashkir Diacritics Restorer v1.2

A compact, local-first ONNX model that restores Bashkir-specific letters in partially undiacritized text while deliberately preserving foreign and technical tokens.

Overview

This is a character-level Bashkir diacritics restorer for the nine Bashkir letters: Ә Ғ Ҙ Ҡ Ң Ө Ҫ Ү Һ. It combines a 1.18M-parameter ConvBiGRU model with a conservative safety gate. The neural model is self-contained and runs with only model.onnx, vocab.json and substitution_map.json. The default runtime does not require a lexicon, KenLM, an internet connection, or a GPU.

Contents

Files and Configurations

The public package intentionally excludes the derived lexicon.json and the optional phrase-level KenLM model. This keeps the release self-contained and avoids redistributing a corpus-derived lexical resource. A research/full variant with optional components may be requested from the author through the Hugging Face profile.

Model Architecture

The model predicts a character distribution at each input position. It uses parallel 1D convolution features (kernel sizes 3 and 5) followed by a two-layer bidirectional GRU with hidden size 192. A substitution whitelist permits only valid base-letter to Bashkir-letter changes; it cannot invent arbitrary symbols.

Examples

Method

input text → token safety gate → character ONNX predictions → confidence threshold
    → restored text

• —Safety gate: skips URLs, email-like fragments, Roman numerals, ordinary Latin tokens, common Russian function words and unknown foreign quoted text.
• —Controlled replacements: only observed base-to-specific-letter substitutions from substitution_map.json can be applied.
• —Existing diacritics: already typed Bashkir-specific letters are retained.

Benchmark

The original held-out evaluation reports sentence exact match of 93.42% and Bashkir-specific-letter F1 of 99.67%. The public standalone release — without both lexicon.json and KenLM — reaches 87.8% restoration exact match and 97.6% untouched clean input on a mixed 1,000-item safety check. That check includes foreign and technical strings that the v1.2 safety gate intentionally leaves unchanged. It also uses automatically generated targets, so individual disagreements can include noisy or historical source forms. See `META.json` for the machine-readable record.

Component Smoke Check

A 15-item behavioural check, ordered from simple word restoration through mixed-script input, casing, ambiguity and protected foreign text, was run with three local configurations. This is a transparent integration check, not a representative benchmark.

The context-free form кул is deliberately retained in all modes: it can mean more than one Bashkir word, and this release does not guess without a phrase-level language model.

On this automatic test the lexical component changes the overall result by 1.3 percentage points. It is most useful for known forms and mixed-script keyboard input; its impact on real-world usage should be measured separately with a human-checked test set.

Quality and Use

This release is suitable for local text cleanup, input-method suggestions and post-processing of Bashkir text. Its small ONNX model is practical on CPU-only systems. For a keyboard, call it only for the current token or short composing text, and present a candidate rather than silently replacing uncertain text.

Limitations

• —This is a restoration model, not a general spellchecker or text generator.
• —A text with no Bashkir lexical evidence may be intentionally left unchanged.
• —Without the optional context language model, homographs can remain ambiguous.
• —The model is trained for Bashkir Cyrillic; code-mixed, dialectal, novel and OCR-noisy forms need application-level review.
• —Quality scores do not establish linguistic normativity for every output.

Related Resources

• —Bashkir FastText Embeddings — compact subword embeddings for similar-form lookup and Bashkir lexical signals.
• —Bashkir Word N-gram Index — phrase-level statistics for optional context-aware candidate ranking in keyboards.

Usage

Install the two runtime dependencies:

pip install -r requirements.txt

Run the included example from this release directory:

python example.py

Or use the runtime directly:

from pathlib import Path
from runtime import BashkirDiacriticsRestorer

restorer = BashkirDiacriticsRestorer(Path("path/to/bashkir-diacritics-restorer"))
print(restorer.restore("мин башкортса хойлэшэм"))
# мин башҡортса һөйләшәм

The default threshold is 0.40. Raise it to make neural substitutions more conservative, for example restorer.restore(text, min_confidence=0.60).

License

Distributed under the CC BY 4.0 license. The release contains model artifacts and derived substitution rules, not source texts. Upstream source licenses and attribution requirements still apply to the underlying materials.

Citation

@software{failed09_bashkir_diacritics_restorer_2026,
  title = {Bashkir Diacritics Restorer v1.2},
  author = {failed09},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/failed09/bashkir-diacritics-restorer},
  note = {Compact ONNX model for safe Bashkir diacritics restoration}
}

Open Bashkir Data and Sources 🐝

This release is part of an open-source effort to support the development, preservation and practical use of the Bashkir language. Other related models, datasets and tools are available on the author's Hugging Face profile.

The author does not claim ownership or authorship of the source texts or other materials used to derive this release; rights and licensing remain with the original authors, publishers and dataset providers. Source texts are not redistributed in this repository, so users should follow the licenses and attribution requirements of the relevant upstream resources.