failed09/bashkir-lid

1---2language:3- ba4license: apache-2.05pretty_name: Bashkir LID (Binary BA vs NON_BA)6library_name: onnxruntime7pipeline_tag: text-classification8tags:9- bashkir10- language-identification11- text-classification12- onnx13- onnxruntime14- low-resource15---16 17# Bashkir LID (Binary BA vs NON_BA)18 19> Compact ONNX language gate that decides whether text is Bashkir, for corpus20> filtering and local CPU inference.21 22## Overview23 24A compact binary language gate for Bashkir (`ba` vs `non_ba`), designed for25corpus filtering, OCR triage and local applications that need lightweight CPU26inference. It was trained with hard negatives from sister Turkic languages,27Russian and technical/noisy text, with special attention to Bashkir/Tatar28separation.29 30| At a glance | |31| --- | --- |32| Task | Binary language identification (`ba` vs `non_ba`) |33| Default artifact | `model.onnx` + `lid.py` |34| Source | Bashkir, Turkic, Russian and technical text (not redistributed) |35| Version / license | v3 / Apache-2.0 |36 37## Contents38 39### Files and Configurations40 41| File | Purpose |42| --- | --- |43| `model.onnx` | Exported classifier graph and weights (opset 17) |44| `vectorizer.json` | Dual-channel character and word feature vocabularies |45| `lid.py` | Portable ONNX Runtime adapter |46| `config.json` | Runtime contract |47| `META.json` | Model passport, validation results and artifact hashes |48| `__init__.py` | Package marker |49| `SHA256SUMS` | Release checksums |50 51Training texts and the original `joblib` pipeline are not distributed.52 53### Model Architecture54 55| Property | Description |56| --- | --- |57| Task | Binary language identification: `ba` or `non_ba` |58| Features | Dual-channel TF-IDF: `char_wb` (2–5) + `word` (1–2), 240,000 features |59| Classifier | Linear SGD classifier exported to ONNX with Sigmoid normalization |60| Runtime | ONNX Runtime on CPU |61| Preprocessing | Unicode lowercase and dual-channel sparse features, provided by `lid.py` |62| Outputs | Class labels (`ba`, `non_ba`) and probabilities |63 64The ONNX graph takes sparse feature tensors (`char_ids`, `char_counts`,65`word_ids`, `word_counts`); the adapter handles preprocessing, batching and66checksum validation. No custom ONNX operators or Transformers installation are67required.68 69### Examples70 71| Text | Prediction | `ba` probability |72| --- | --- | ---: |73| `Мин башҡорт телен яратам.` | `ba` | 0.998 |74| `Бары бала бара.` | `ba` | 0.927 |75| `Сегодня хорошая погода.` | `non_ba` | 0.044 |76| `SELECT id FROM users WHERE active = TRUE;` | `non_ba` | 0.011 |77 78Note the second row: valid Bashkir text does not always contain79Bashkir-specific letters.80 81## Method82 83The model is a dual-channel linear classifier: TF-IDF `char_wb` (2–5) and word84(1–2) features are fused and passed to an SGD classifier exported to ONNX. It was85trained with hard negatives from sister Turkic languages, Russian and86technical/noisy text. Training texts are not distributed in this repository.87 88### Evaluation89 90On a weak-label 4,000-sentence benchmark the model reached **93.25% binary91accuracy**: at the default 0.50 threshold it retained 730/1,000 Bashkir examples92and rejected 3,000/3,000 non-Bashkir examples. A hand-authored adversarial suite93passed 26/26 (Tatar, Kazakh, Chuvash, Russian with Bashkir inserts, code and94noise), and an independent 100-example stress set reached 97/100. Labels are weak95or manually constructed, not an expert-reviewed Gold benchmark; exact numbers are96recorded in [`META.json`](META.json).97 98### Operating thresholds99 100`predict()` selects `ba` when its probability is at least 0.50. Measured101operating points (from [`META.json`](META.json)):102 103| BA probability threshold | BA recall | Non-BA false acceptance |104| ---: | ---: | ---: |105| 0.50 | 73.0% (730/1,000) | 0/3,000 |106| 0.40 | 78.3% (783/1,000) | 2/3,000 |107| 0.35 | 80.8% (808/1,000) | 4/3,000 |108 109Use `0.50` for the strictest rejection of non-Bashkir text. Use `0.35–0.40` when110preserving more Bashkir text matters, preferably with dictionary and quality111checks. The price of the strict gate is false rejection of some valid Bashkir112sentences, especially short or BA/Tatar-like sentences.113 114## Quality and Use115 116This is a binary gate, not a general multilingual language detector. It does not117identify which non-BA language was found. Tatar and other languages, short118strings, names, OCR artifacts and mixed-language text can be ambiguous.119 120### Limitations121 122- Binary decision only: `ba` vs `non_ba`, with no language label for rejects.123- Weak-label benchmark; not an expert-reviewed Gold evaluation.124- Confidence values are model scores, not calibrated human probabilities.125- Empty strings, OCR fragments and technical noise should be handled with126  explicit pipeline rules.127- For high-recall corpus construction, combine with the multiclass LID,128  dictionaries, quality rules and human review.129 130## Related Resources131 132- [Bashkir Multiclass LID](https://huggingface.co/failed09/bashkir-lid-multiclass) —133  distinguishes `ba`, `tt`, `ru` and `other`; use it when you need the language134  label instead of a pass/reject gate.135 136## Usage137 138```bash139pip install huggingface_hub onnxruntime numpy140```141 142```python143import sys144from huggingface_hub import snapshot_download145 146model_dir = snapshot_download(147    "failed09/bashkir-lid",148    allow_patterns=[149        "lid.py", "config.json", "META.json", "model.onnx", "vectorizer.json"150    ],151    revision="main",  # pin a reviewed commit for reproducible deployments152)153sys.path.insert(0, model_dir)154from lid import LanguageIdentifier155 156lid = LanguageIdentifier(model_dir=model_dir)157texts = ["Мин башҡорт телен яратам.", "Сегодня хорошая погода."]158print(lid.predict(texts).tolist())159print(lid.predict_proba(texts))160print(lid.classes.tolist())  # probability-column order161```162 163After downloading, inference runs locally without network access. For a local164checkout with the model files alongside `lid.py`, `LanguageIdentifier()` also165works without extra arguments.166 167## License168 169The code and model export are released under the170[Apache-2.0 license](https://www.apache.org/licenses/LICENSE-2.0). Training texts171are not included in this repository.172 173## Citation174 175```bibtex176@software{failed09_bashkir_lid_2026,177  title = {Bashkir LID},178  author = {failed09},179  year = {2026},180  publisher = {Hugging Face},181  url = {https://huggingface.co/failed09/bashkir-lid},182  note = {Compact binary Bashkir language gate}183}184```185 186## Open Bashkir Data and Sources 🐝187 188This release is part of an open-source effort to support the development,189preservation and practical use of the Bashkir language. Other related models,190datasets and tools are available on the author's Hugging Face profile.191 192The author does not claim ownership or authorship of the source texts or other193materials used to derive this release; rights and licensing remain with the194original authors, publishers and dataset providers. Source texts are not195redistributed in this repository, so users should follow the licenses and196attribution requirements of the relevant upstream resources.197