syamjithnk/arabic-corpus-audit

Arabic Corpus Integrity Audit

Author: Syamjith NK Date: 9 September 2026 · corrected 13 September 2026 Tool: `arabic-lint` 0.5.0

Correction, 13 September 2026. An earlier version of this card said the labels in Yousefmd/arabic_ocr_dataset were stored in visual order, and called that the full reshape + bidi signature. That was wrong. Only the shaping step ran; the words are in logical order and plain NFKC recovers them. What was measured, and stands, is that the labels store presentation forms. Full detail, and how the error happened, under Corrections.

The question

The arabic_reshaper + python-bidi recipe converts Arabic to presentation forms and visual order. Either half can run on its own, and this audit detects only the first: stored presentation forms. It is correct for renderers that do no shaping, and it silently corrupts text on ones that do. It appears in 3,168 indexed Python files on GitHub.

If that recipe ever ran upstream of a training corpus, every model trained or evaluated on it learned from text no Arabic reader would accept. Nobody had measured whether it did.

What is in here

One row per dataset, for 341 public Arabic datasets on the Hub.

Severity, and why it decides what the numbers mean

A reshaped run means presentation forms were written to the file, which is what arabic_reshaper.reshape() produces. It does not by itself establish that bidi.algorithm.get_display() also ran. Word order has to be measured separately, and in the one dataset where it was measured the bidi step had not run. See Corrections.

Counting findings without this distinction is misleading, because the two have different causes and different fixes: a stray glyph is a character to correct, a reshaped run means the pipeline that wrote the file is the problem and every other file it touched needs checking.

Across the whole audit: 515 stray spans, 9 partial, 108 reshaped — and all 108 reshaped spans are in a single dataset.

Headline result

276 datasets readable · 26,318 rows · 119,517 text fields · 21 with any finding · 1 with a presentation form in every sampled field.

`Yousefmd/arabic_ocr_dataset` stores presentation forms in every sampled label — 1,000 of 1,000 labels, sampled through ten windows spanning the whole 20,000-row split, 3 to 50 presentation forms per label (median 24), INITIAL/MEDIAL/FINAL/ISOLATED forms together.

Only the shaping half of the recipe ran there. The words are in normal logical order, not visual order, so unicodedata.normalize("NFKC", text) alone recovers the text correctly and no reversal is needed. Measured two ways: on 5,423 sampled words carrying more than one presentation form, 0 begin with a FINAL form, which is what visual order looks like in memory; and arabic_reshaper.reshape(NFKC(stored)) == stored for 1,000 of 1,000 labels, so the stored string is exactly reshape() output with get_display() never applied. An earlier version of this card claimed visual order. That was wrong. See Corrections. Both figures are re-derivable with the shipped verify_yousefmd.py, which asks arabic-lint's own is_presentation_form() rather than matching Unicode names — a name test for ARABIC LETTER ... FORM drops every lam-alef ligature, which is the one character this measurement is most sensitive to.

It matters because the corrupted field is the label of an OCR set: a model trained on it learns to emit presentation forms rather than plain letters, so its output will not compare equal to ordinary Arabic even when the recognition is right. This is the propagation path from a rendering bug into model weights, observed rather than argued for.

⚠️ Its scale, stated plainly: 17 downloads, 0 likes. This proves the mechanism reaches training data. It is not evidence that widely-used Arabic corpora are affected.

The other 20 flagged datasets had no reshaped runs at all — 515 stray and 9 partial spans, most often U+FE91 BEH INITIAL FORM, from OCR and copy-paste.

Measured, and it is worse than "different tokens". On tokenizers without a Unicode normalisation step, a word containing one stray presentation form is not split, it is replaced by the unknown token. Every character of meaning is discarded before the model sees it. One corrupted letter out of four is enough: بيت becomes [UNK].

The irony is that the two Arabic-capable BERTs destroy the word while the multilingual SentencePiece model is immune, because XLM-R normalises and they do not. Re-runnable: tokenizer_cost.py, tokenizer files only, no weights.

What this does not establish

• —65 datasets could not be read (server errors, no viewer, gated). The viewer is most often unavailable on the largest corpora, so this covers the mid and small end well and the head of the distribution poorly. No claim here reaches web-scale pretraining data.
• —100 rows per dataset cannot find a defect confined to a rare subset.
• —Presence of a presentation form is evidence of stored glyph forms, not proof of the specific recipe. The distinction is why severity is reported separately.
• —A presentation form says nothing about word order. Shaping and bidi are two separate steps and either can run without the other. results.jsonl records stored glyph forms only; nothing in this dataset measures visual order, and no row here should be read as a claim about it.

Corrections

Findings here are about other people's public work, so there has to be a way to contest them and a record of when one was contested successfully.

To dispute a finding: open a discussion on this dataset, or reply in the thread on your own repository if one was opened there. A dataset id and one row that you believe was scored wrongly is enough; you do not need to diagnose the bug.

What is committed to:

• —An acknowledgement within 3 days, whether or not the dispute has been assessed yet.
• —A re-measurement against the live rows, not against these stored results, within 7 days.
• —If a finding is wrong it is corrected in place, here and in every other place it was published, and the old claim is left visible with a note rather than deleted. Anything published elsewhere about the specific repository is corrected in the same thread it was raised in.
• —The error is written up in the log below, including what caused it. A correction that hides how it happened teaches nobody anything.

Correction log

2026-09-13 — `Yousefmd/arabic_ocr_dataset` is not stored in visual order. The card, the findings write-up and a discussion opened on that repository all stated that the labels had been through arabic_reshaper.reshape() and bidi.algorithm.get_display(), and called that "the recipe's complete signature". Only the shaping step ran. Verified two ways against the live rows: plain unicodedata.normalize("NFKC", stored) returns correct readable Arabic in logical order with no reversal, and of 5,423 multi-form words 0 begin with a FINAL form. The practical consequence is that the repair advice given was wrong in the worse direction: it warned that recovery needed more than NFKC because of the lam-alef ligature, when for this data NFKC alone is a complete and safe repair. The counts were sound; the mechanism inferred from them was not. Also corrected in the same pass: "23 to 29 presentation forms per label" (real range 3 to 50, median 24) and the headline word "corrupted" where "carries stored presentation forms" is what was actually measured.

Cause, stated plainly. The detector reports stored glyph forms. Visual order is a separate property and was never measured, only assumed, because the two steps are usually copied together. The fix is in the tool's own vocabulary: a reshaped severity now means a shaping pass ran, and nothing more.

A false positive worth knowing about

The first run reported 35.6% of an Islamic heritage OCR corpus as corrupted. Every hit was a false positive. Arabic Presentation Forms-A is interleaved: the ornate parentheses ﴾ ﴿ enclosing a Quranic quotation are U+FD3E/U+FD3F, inside the positional-form range, with honorific ligatures just above. Treating the block as uniform accuses correct heritage text.

Fixed in arabic-lint 0.5.0 by deriving the class from the Unicode character name rather than tabulated ranges. Anyone auditing Arabic text should check this before reporting a number.

Reproducing

pip install arabic-lint
python3 audit_all.py          # https://github.com/Syamjith-NK/arabic-lint

Citation

Archived with a DOI on Zenodo, so the exact version behind these numbers stays citable even as this card is updated: [10.5281/zenodo.22733934](https://doi.org/10.5281/zenodo.22733934)

@dataset{syamjith2026arabicaudit,
  title     = {An audit of presentation-form corruption in 341 public Arabic datasets},
  author    = {NK, Syamjith},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.22733934},
  url       = {https://doi.org/10.5281/zenodo.22733934}
}