kurdish-tech/KurdishCorpus-clean

The Largest Documented Kurmanji and Multi-Dialect Kurdish Dataset — v1.1

A large-scale, multi-dialect Kurdish text corpus prioritizing native Kurmanji (Northern Kurdish) fluency, with substantial Sorani (Central Kurdish) and a Zazaki baseline. Built for language modeling, tokenizer training, and general-purpose Kurdish NLP.

This release contains only openly-licensed or presumptively-free redistributable content. A parallel research-tier subset (copyrighted commercial news/broadcast scrapes and a small set of copyrighted private books — see Licensing below) exists locally for internal training use but is deliberately excluded from this public release.

Dataset Statistics (v1.1, open-release)

v1.1 adds ~371,500 Kurmanji documents (+167.1M words) over v1.0, almost entirely from two new sources (see Sources below): kurdish-web, a CC-BY-4.0 Kurdish/Zaza web-crawl corpus (Kurmanji portion only — this project deliberately ingests Kurmanji-only from new multi-dialect sources by default, leaving that source's Sorani/Zazaki configs uncollected), and kurdish-theses, a small CC-BY-4.0 academic-thesis corpus. Sorani and Zazaki totals are byte-for-byte unchanged from v1.0, confirming this update is Kurmanji-only as intended.

Exact token counts (verified, reproducible)

The word counts above are whitespace-token counts computed at extraction time, not subword/model tokens. For an exact figure, every document in this exact release file was tokenized with OpenAI's tiktoken (cl100k_base encoding) on 2026-07-17:

Sorani's Arabic script is markedly less token-efficient under generic BPE vocabularies than Kurmanji's Latin script (roughly 7 tokens/word vs. ~2.8), so its token share overstates its real proportion relative to the word counts above; a custom Kurdish tokenizer would close most of this gap.

This count is exact and reproducible, not an estimate — run python scripts/count_exact_tokens.py data/final/corpus_v2_open_release.jsonl yourself against the released file to verify it. It is also, as far as we know, the only published Kurdish corpus that discloses a per-dialect token breakdown at all: other large aggregate Kurdish corpora advertise a single multidialectal total token count without stating which tokenizer produced it or how it splits across dialects, which makes independent verification or a fair Kurmanji-specific comparison impossible from their documentation alone.

Sources

The Sources and Licensing sections above are self-contained for this release's purposes; they summarize a longer internal source-verification process (per-source license checks, volume estimates, and the engineering history behind each collection phase) that isn't included in this distribution.

Licensing — why some content is not in this release

This project maintains a two-tier policy: open (the license permits redistribution, included here) and research (copyrighted material collected for internal model training only, never redistributed). The following sources exist in the project's full working corpus but are excluded from this public release:

Note on the Rojava school curriculum specifically: this source is included in the release (see Sources above), on the basis that it is distributed freely by the Rojava Autonomous Administration for educational use. That representation has not been independently verified by this project against a specific copyright statute or license text — the same category of uncertainty as the Kurdistan Parliament legislation above, which is kept research-tier for exactly this reason. If you plan to build on this subset specifically, you may want to do your own diligence before treating it as unambiguously public domain.

If you need the research-tier subset for your own (non-redistributed) training, it is available in the project's working directory; it is not part of this Hugging Face release and should not be re-uploaded elsewhere without independently clearing the licensing questions above.

Data Cleaning & Quality Pipeline

1. 1.Collection — per-source extractors (Wikimedia XML dumps, HPLT/MADLAD/ FineWeb-2/CC-100 web-crawl shards, OPUS parallel corpora, targeted PDF/HTML scrapers) into a common JSONL schema.
2. 2.Exact deduplication — hash-based, at merge time.
3. 3.Near-duplicate removal — MinHash/LSH (Jaccard ≥0.8) across the whole prose corpus, keeper priority favoring more-curated sources.
4. 4.Language identification — GlotLID (fastText), removing documents whose predicted language isn't a Kurdish variant above threshold. Dictionary/curriculum sources are exempted from removal but still fully classified, which enabled the fix below.
5. 5.Gopher-style quality filtering — min/max length, symbol ratio, repetition (line/paragraph/n-gram), stopword presence; dictionary and curriculum sources are exempted from the length/repetition battery (short definitions and formulas would otherwise be wrongly rejected) but still checked for basic non-corruption.
6. 6.Surgical language sanitation (this release's key fix) — during PDF ingestion of the Rojava school curriculum, three Arabic-language subject textbooks (an elementary Arabic-language course, titled "عربي" / "عربي مكون كردي") slipped through alongside ~100 genuinely Kurdish curriculum PDFs. A first attempt to remove them with a hand-built Arabic character/stopword density heuristic backfired badly: because several Arabic letters (ة ث ص ض ط ظ ذ ي) also appear incidentally in ordinary Kurdish text — quoting an Arabic-script proper noun in parentheses (a standard Kurdish-Wikipedia convention, e.g. "Trablûs (bi erebî: طرابلس)"), or a dictionary correctly documenting an Arabic loanword's etymology — the density filter quarantined 17,574 documents when only 3 were actually contaminated, with 92% of the false positives coming from already-verified-clean FineWeb-2 Sorani content. The fix: every document's GlotLID classification (lid_label) was already computed and stored during step 4, just never used for this purpose. Filtering specifically on source == "rojava-curriculum" AND lid_label == "arz_Arab" identified exactly the 3 contaminated documents, with zero false positives — confirmed by manual inspection of all three (subject fields read "عربي" / "عربي مكون كردي") and of the full 102-document curriculum set (the rest carry Kurdish or classification-noise labels, never a genuine Arabic one). This is the recommended lesson for anyone extending this corpus: prefer a model-based whole-document signal already computed by your pipeline over a hand-built character/keyword heuristic for language-purity checks — the latter cannot distinguish "this document is in language X" from "this document merely mentions language X." This quarantine step is re-applied on every pipeline rebuild (new sources go through the same merge → near-dedup → LID → quality → surgical-sanitation sequence); the v1.1 rebuild reconfirmed the identical 3 documents and zero false positives.

Intended Use

Pretraining and fine-tuning language models, embeddings, tokenizers, and other NLP systems for Kurdish (Kurmanji-priority, with Sorani and Zazaki support). Not vetted for toxic/harmful content beyond the quality filters above; downstream users training public-facing systems should apply their own safety filtering.

Citation

If you use this dataset, please cite the source repository and, where applicable, the upstream datasets listed in Sources above (particularly FineWeb-2, MADLAD-400, HPLT, and the Kurdish Wikimedia projects, each of which has its own citation guidance).