erdemKocaogluu/byt5-small-tr-normalizer

byt5-small-tr-normalizer

A fine-tuned byT5-small model that rewrites informal, misspelled, or dialectal Turkish sentences into standard written Turkish.

Typical inputs are the kind of text people actually type in a hurry: dropped diacritics, missing letters, colloquial verb contractions, ASCII-only typing, and regional/dialectal spelling. The model rewrites these into a clean, standard-Turkish sentence while keeping the original meaning and word order.

sicil belgmi alcam            -> Sicil belgimi alacağım
araç lsansı nasl alacaım      -> araç lisansı nasıl alacağım
muhtara gitcem ne götüreyim   -> Muhtara gideceğim ne götüreyim

Model description

• —Base model: google/byt5-small
• —Tokenizer: byte-level (no vocabulary, no BPE merges) — every UTF-8 byte is a token, so the model reasons directly over characters. This makes it naturally robust to the kind of noisy, non-standard spelling this task targets.
• —Task: sequence-to-sequence text normalization (noisy Turkish -> standard Turkish)
• —Input format: prefix the sentence with "düzelt: " before feeding it to the model (see usage below).

How to use

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

model_id = "erdemKocaogluu/byt5-small-tr-normalizer"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)

text = "sicil belgmi alcam"
inputs = tokenizer("düzelt: " + text, return_tensors="pt")
output = model.generate(**inputs, max_length=384, num_beams=1)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Training data

Training pairs come from six public Turkish datasets that already provide noisy/clean sentence pairs (grammar correction, typo validation, text normalization — see Data sources below). No data was synthetically generated for this training run. Four sources are permissively licensed (MIT, Apache-2.0, or CC0-1.0); two are derived from Turkish Wikipedia and are licensed CC BY-SA 3.0 (attribution required — see below).

Data sources

Not every dataset that was evaluated made it into the final training mix — a diacritics-restoration dataset, for example, was tested but excluded after evaluation.

The MIT, Apache-2.0, and CC0-1.0 sources above are acknowledged here per their license terms; their original copyright notices and licenses remain with the linked repositories. The trspell (NoisyWikiTr) and noisy-sentences-dataset sources are built from Turkish Wikipedia text; that text is licensed CC BY-SA 3.0, so it is credited here accordingly: portions derived from Wikipedia contributors, licensed under [CC BY-SA 3.0](https://creativecommons.org/licenses/by-sa/3.0/). This project does not redistribute the original dataset files — only sentence pairs derived from them were used to train the model.

Note on Turkish-OSCAR-GEC: this dataset is built on top of OSCAR, a Common Crawl–derived web text corpus. OSCAR's own documentation states that its maintainers "do not own any of the text from which these data has been extracted" and license only their own metadata/annotations (CC0). The "MIT" tag on Turkish-OSCAR-GEC is the uploader's own declaration and does not itself establish clear rights over the underlying crawled text. This is disclosed here so downstream users can make their own informed assessment; it reflects a broader, industry-wide open question around web-scraped training corpora rather than an issue specific to this dataset.

OSCAR's own terms note an exception for text-and-data-mining (TDM) and research use; the way this data was used here — extracting statistical spelling/grammar-correction patterns during training, not reproducing or redistributing OSCAR's source sentences — fits that framing. The model is also not a free-text generator: it transforms a sentence the user supplies, so its output is structurally anchored to that input rather than free to reproduce memorized training text, which limits (though does not eliminate) the risk of verbatim reproduction of training data.

Training procedure

During training, checkpoints were evaluated periodically on a held-out validation split and ranked with a combined selection score that balances two things: how well a checkpoint fixes noisy sentences, and how well it leaves already-correct sentences untouched (to discourage over-correction). The checkpoint used here is the highest-scoring one from that process, and its weights were verified by hash before being published.

Q_good          = 1 - (0.6 * CER + 0.4 * WER)       # how well it fixes noisy sentences
unchanged_rate  = pred == source on clean sentences  # how well it leaves correct text alone
selection_score = 0.7 * Q_good + 0.3 * unchanged_rate

Training curves

[image]

[image]

[image]

[image]

[image]

[image]

Evaluation

Held-out gold sets (naturally occurring, human-written Turkish — never seen during training)

"copy" = do-nothing baseline (source passed through unchanged).

The model roughly halves the word error rate compared to a "do nothing" baseline on both sets, while rarely damaging text that was already correct. On boun, note that model CER is slightly higher than the copy baseline's CER even though WER improves substantially — WER and CER don't always move together, since a small number of larger character-level edits can outweigh many small word-level fixes.

[image]

[image]

Sources:

• —tweets — Köksal, A. T., Bozal, Ö., Yürekli, E., & Gezici, G. (2020). #Turki$hTweets: A Benchmark Dataset for Turkish Text Correction. Findings of ACL: EMNLP 2020, pages 4190–4198.
• —boun — Kara, A., Marouf Sofian, F., Bond, A., & Şahin, G. (2023). GECTurk: Grammatical Error Correction and Detection Dataset for Turkish. Findings of ACL: IJCNLP-AACL 2023, pages 278–290.

The tweets set was lightly cleaned before evaluation — social-media artifacts (@mentions, hashtags, emoji) were stripped from both source and reference, and rows the original dataset flagged as foreign-language or neologism were excluded.

Limitations

• —The model generalizes very well to mechanical, deterministic corrections (like deasciification), where the rule mapping is unambiguous.
• —It is comparatively weaker on organic, human-style noise that doesn't closely match the synthetic noise patterns it was trained on, and on long, multi-clause informal sentences.
• —It corrects spelling, spacing, and word-internal errors; it does not reorder words or rewrite sentence structure.
• —Inference is CPU-friendly but on Apple Silicon MPS the model can occasionally produce degenerate output; CPU or CUDA is recommended.

License

These fine-tuned model weights are released under CC BY-NC-SA 4.0 (non-commercial use, attribution required, share-alike). Note this differs from the base model: google/byt5-small itself remains available under its own Apache 2.0 license. This license applies to how these fine-tuned weights are released; it is separate from the licenses of the training data sources listed above under Data sources, consistent with common practice for models trained on mixed-license corpora.