SultanR/AraMix-Native

AraMix-Native

A native-Arabic-filtered version of `AdaMLLab/AraMix` (minhash_deduped), derived from `SultanR/AraMix-Translation-Scores`: machine-translated and garbled-MT documents removed, 162,887,010 rows kept of 178,883,241 (91.06%). All columns preserved.

Filter rules

A document is kept iff all of:

1. 1.mmbert_translated_score < 0.1, or a classical-text rescue: diacritic (tashkeel) ratio ≥ 0.02 over Arabic letters and ≥ 3 distinct diacritic classes (fully/partially vocalized classical prose, Qur'anic text, poetry — verified to exclude MT that carries only tanwīn).
2. 2.No China-catalog MT pattern (machinery/manufacturer boilerplate — a large MT class the classifier scores ≈ 0 because it is far more broken than the fluent backtranslations it was trained on).
3. 3.Arabic stopword cadence ≥ 0.035 (orthography-normalized, و/ف proclitics stripped, tatweel removed) — drops keyword-stuffed/garbled pages.
4. 4.≥ 10 mojibake characters → dropped; < 20 Arabic words → dropped.

Known residuals (manually verified by sampling)

• —Fluent-but-slightly-broken MT that the classifier scores < 0.1 (e.g. pharma pages mixing Latin chemical terms) remains — est. ~1% of kept rows worst-case.
• —The garble gate wrongly drops a small amount of name-dense prose (sports line-ups, protocol news), est. ≤ 0.5% of the corpus.
• —The MT classifier (mmbert_translated_score, see the scores dataset card) was trained on gemma-4 / Seed-X / Llama-3.1 backtranslations; MT from very different systems (e.g. older SMT) may not be flagged.