algerian-nlp/algerian-darja-corpus

Algerian Darja Corpus

11,151 long-form conversational transcripts in Algerian Darja for language modeling of real spoken Algerian, by Kamel Touati (Independent AI Researcher, Algiers, ORCID 0009-0000-5330-2123, Hugging Face touati-kamel), mirrored on the Algerian NLP Collective. Counted 2026-09-17 via the Hub datasets-server (/info?dataset=algerian-nlp/algerian-darja-corpus: 11,151 train rows) and re-counted row-by-row with datasets streaming (load_dataset("algerian-nlp/algerian-darja-corpus", split="train", streaming=True): 11,151 rows).

The default config answers: what does unscripted Algerian conversation look like at document scale, with French and English code-switching intact.

from datasets import load_dataset

ds = load_dataset("algerian-nlp/algerian-darja-corpus", "default", split="train")

default — long-form conversational transcripts

11,151 rows, all in train. No dev or test split ships with this repo.

The gap this config fixes, with the measurement: Algerian Darja language modeling needs multi-thousand-word documents, not isolated sentences. 6,179 of 11,151 rows (55.4%, counted by streaming filter int(word_count) > 1000) exceed 1,000 words per the row-level word_count field; the author's paper reports 34,336,366 words and 186,106,376 characters (~53M subword tokens) with median document 1,116 words, mean 3,079.2 (σ 6,568.8), max 104,328, and type-token ratio 0.0333 (academic_paper/paper.tex in this repo).

No row id field exists: rows join to nothing and have no stable key — key on row order only, which breaks across re-downloads. The text is normalised to Unicode NFKC with tashkeel stripped (shaddah kept), tatweel stripped, control characters and raw URLs removed, per the card's preprocessing section. What breaks if ignored: word_count will not reproduce exactly under a different tokenizer or NFC normalisation — recount before reporting corpus word totals.

Scoring protocol

Language-modeling evaluation only: report perplexity with the exact tokenizer, stride, and context length beside the number. Perplexity on this set is NOT comparable to perplexity on sentence-level Darija sets. Unmeasured: no baseline perplexity has been scored by us on this protocol.

Results

Unmeasured. No baseline results scored by us in the protocol above.

Usage

default: unscripted long-form Algerian conversation for language modeling.

from datasets import load_dataset

ds = load_dataset("algerian-nlp/algerian-darja-corpus", "default", split="train")
row = ds[0]
# row keys: text, word_count, character_count
# row["word_count"]: 7764 — row["character_count"]: 47061
# row["text"] opens: "On m'a propos\u00e9 des tickets \u00e0 2 millions de dollars. ..."

# Long documents only (measured 2026-09-17, streaming, 11,151 rows scanned):
long_docs = ds.filter(lambda row: int(row["word_count"]) > 1000)
len(long_docs)  # 6179

What this dataset does NOT contain — and what breaks if you assume it does. No speaker labels (stripped at build), no timestamps, no per-row source id, licence, or retrieval date despite the collective rule requiring them, and no split beyond train. Do not assume one row equals one video or one speaker turn — segmentation is by transcript chunk, undocumented.

Script & code-switching distribution

Author-measured over all 11,151 documents (academic_paper/paper.tex, Table 1 — the author's measurement, not ours):

At character level the author reports 145,678,766 Arabic-script characters (78.28%) against 3,328,619 Latin characters (1.79%). No independent recount was run by us.

Provenance and limits

• —The default config derives from YouTube podcast transcripts (60+ channels listed in the original card body) transcribed with Google Gemini speech-to-text, plus three oddadmix Algerian audio collections, and inherits CC-BY-4.0. Transcript sentences are the creators', not ours.
• —Per-row provenance is missing: rows carry no source id, no licence, and no retrieval date. The YouTube channels of origin are named only at corpus level.
• —Filtration audit. Author-documented, not independently verified: a 15-stage pipeline (subtitle timestamp and acoustic-tag stripping, multi-scale n-gram dedup at n ≤ 30, elongation collapse, foreign-script filtering, NFKC normalisation, tashkeel stripping with shaddah kept, dialect-marker filtering) per academic_paper/paper.tex. The author's claim that 100% of retained documents contain genuine Darja is the author's claim, not our measurement.
• —Bounded, not cleared. Profanity, personal data, and copyrighted show content cannot be excluded: web and social text is used as published. The author's PII handling is undocumented.
• —Out of scope. Moroccan and Tunisian dialect separation was not measured; eastern and Saharan Algerian coverage is unstated and would corrupt any claim of national coverage.
• —Decontamination of any training corpus against this dataset is unmeasured.

Citation

@misc{algerian_nlp_algerian_darja_corpus,
  title  = {Algerian Darja Corpus: 11,151 long-form conversational transcripts},
  author = {Touati, Kamel and Algerian NLP Collective},
  year   = {2026},
  doi    = {10.5281/zenodo.22723182},
  url    = {https://huggingface.co/datasets/algerian-nlp/algerian-darja-corpus}
}

The author's own BibTeX declares version 2.1.0 and titles the work "Algerian Darja Corpus: A 34-Million-Word Long-Form Conversational Dataset for Dialectal Language Modeling"; the full paper ships in this repo under academic_paper/paper.tex.

Upstream: three oddadmix Algerian audio collections (arabic-audio-collection-algerian-kahwa-postcast, arabic-audio-collection-algerian-loubna-stories, arabic-audio-collection-algerian-rawi) — cite their cards alongside this one.

Licence

CC-BY-4.0, inherited from the source transcripts. Attribution is obligatory: credit the dataset and link the licence when you share or adapt it.