razhan/script_normalization_ckb
Script normalization CKB — noisy → standard Sorani (script_normalization_ckb) Nearly 6M sentence pairs: the text column holds Central Kurdish written with non-standard or distorted characters, summary holds the same sentence in standard Sorani orthography. Use it for text normalisation, spell correction, or to learn the character-level mapping rules. At a glance Rows 5,997,025 — train 5,330,689 / test 666,336 Columns text (noisy input), summary… See the full description on the dataset page: https://huggingface.co/datasets/razhan/script_normalization_ckb.
Script normalization CKB — noisy → standard Sorani (scriptnormalizationckb)
Nearly 6M sentence pairs: the text column holds Central Kurdish written with non-standard or distorted characters, summary holds the same sentence in standard Sorani orthography. Use it for text normalisation, spell correction, or to learn the character-level mapping rules.
At a glance
Naming note: despite the column name,summaryis not a summary — it is the normalised rewrite oftext. Both sides carry the same content.
How the two sides differ
Measured on 1,000 rows of the test split:
Which characters get fixed (rows affected out of 1,000):
Example pair:
text : يابان بؤ خؤي بش ئه و قه يرانه طه نگوجه له مه ي ئابووري هه ره فراواني ...
summary : یابان بۆ خۆی پێش ئەو قەیڕانە تەنگوچەلەمەی ئابووری هەرەفراوانی ...Some rows are far noisier than character substitution:
text : اهي اهگهر پئشمهرگه نهبايه حكمهطي بهعث عثهي پاقلاه ...
summary : ئەی ئەگەر پێشمەرگە نەبووایە حکومەتی بەعس عوسەی پاقلاوەو ...Usage
from datasets import load_dataset
ds = load_dataset("razhan/script_normalization_ckb", split="test")
print(ds[0]["text"]) # noisy
print(ds[0]["summary"]) # standard
# fine-tune any encoder-decoder for normalisation, or mine substitution rules:
pairs = ds.select(range(200_000))Known limitations
- The corpus is synthetic in construction: noise was introduced into clean text, so the error distribution reflects the generator, not real user typos.
summaryis the target and the column name is misleading (left in place for backwards compatibility).- Only character/orthography-level repair is targeted; it does not correct grammar or word choice, and heavily distorted rows can still normalise to a different token sequence.
- Rows are single sentences — no document context.
Related
razhan/riste— line-by-line Sorani sentences (a likely source pool).razhan/kteb,razhan/kteb-dataset— book text in mixed orthography.