abdelhaqueidali/Tashelhit-Tifinagh-Latin-Parallel-Tatoeba-Dataset

Dataset Card for Tachelhit Latin-to-Tifinagh Parallel Corpus

This dataset provides a parallel corpus of the Tachelhit language ($\text{Tacelḥit}$ / $\text{Tamazigt}$), pairing native Latin-based orthography with automated Amazigh script transliterations. It is built by processing clean source sentences through an algorithmic engine designed to handle phonetic mappings, manage contextual schwa distributions, and safeguard acronyms and foreign proper names.

Dataset Details

Dataset Description

• —Curated by: Abdelhaque Id Ali
• —Language(s) (NLP): Tachelhit (Latin script and Amazigh script)
• —License: CC-BY 2.0 (Inherited from the Tatoeba source corpus)

Dataset Sources

• —Source Project: Tatoeba Translation Corpus

Uses

Direct Use

• —Training and fine-tuning machine translation models and sequence-to-sequence tokenizers for Southwestern Amazigh dialects.
• —Building automated, rule-based transliteration tools and orthographic normalizers.

Out-of-Scope Use

This dataset relies on literal character-to-character mapping transformations. It is not designed to resolve advanced phonetic mutations that are unrepresented in standard Latin orthographies, nor is it optimized to automatically translate non-adapted loanwords beyond defined exception parameters.

Dataset Structure

The dataset comprises two distinct configuration subsets formatted as Tab-Separated Values (.tsv) files:

1. Monolingual Subset (shi_tifinagh)

• —id: Unique sentence identifier inherited from the Tatoeba database.
• —shi_tfng: The processed Amazigh alphabet transliteration column.
• —shi_latn: The original source Tachelhit sentence in standard Latin script.

2. English Parallel Subset (en_shi_tifinagh)

• —id: Unique sentence identifier.
• —shi_tfng: The processed Amazigh alphabet transliteration column.
• —shi_latn: The source Tachelhit sentence in standard Latin script.
• —en: The corresponding English translation string (extracted and isolated from the raw source pairing).

Dataset Creation

Curation Rationale

While Tachelhit is frequently written in a modified Latin alphabet across digital platforms, the official Amazigh alphabet is standard for educational and cultural frameworks. This dataset establishes an explicit algorithmic bridge between these scripts, offering structured pairs for computational research in low-resource settings.

Source Data

Data Collection and Processing

The source text lines are compiled from the Tachelhit (shi) subset of the Tatoeba repository. The text is normalized using a specialized Python pipeline applying structural and linguistic constraints:

1. Character Map & Digraph Priority

The engine maps Latin graphemes directly to the 33 official IRCAM standardized alphabet characters. Multi-character digraph entries are evaluated first to prevent breakdown into incorrect atomic tokens:

• —Vocalic Digraph: ou (and its casing variants Ou, oU, OU) maps explicitly to the labial vowel ⵓ.
• —Standard Labialization: The labialized marker ʷ is mapped directly to ⵯ.
• —Standard Alternations: Global adjustments ensure mapping across specific vowels and consonants (e.g., o/O => ⵓ, ɛ/Ɛ => ⵄ, c/C => ⵛ, ɣ/Ɣ => ⵖ).

2. Hyphen Deletion

Unlike complex morphological affix-merging systems, all structural or punctuation dashes (-) are globally replaced with standard empty spaces before tokenization to treat word elements uniformly.

3. Contextual Schwa (ⴻ) Optimization

The pipeline treats the schwa (ⴻ) dynamically based on adjacent phonetic environments:

• —Internal Deletion: The engine strips the schwa (ⴻ) if it sits between two different characters (e.g., removing transitional vowel artifacts).
• —Gemination Preservation: The schwa is strictly preserved if the surrounding left and right characters are identical, protecting the vocalic separation of geminated consonants.

4. Text Preservation & Acronym Guardrails

To prevent the corruption of foreign or structural tokens, words are exempted from transliteration and kept in Latin if:

• —The word is fully capitalized and contains more than one character (e.g., acronyms like MRI or M.R.I.).
• —The word consists of a single capital letter, with the absolute exception of the high-frequency English pronoun I (which is permitted to process since some contributors may have English auto correct on which usually turns an i into I even on an Amazigh keybaord).

5. Manual Override Exceptions

Ubiquitous name patterns common in the source text bypass standard rule mechanics and map directly to defined, phonetic target outputs:

• —Tom / tom / ṭom / ⵟom / tum / ṭum => ⵟⵓⵎ
• —Mary / mary => ⵎⴰⵔⵉ

Who are the source data producers?

The original data vectors are crowdsourced, verified, and translated by independent contributors and native speakers on the open-source Tatoeba platform.

Bias, Risks, and Limitations

• —Literal Orthographic Dependency: The script assumes the source Latin text accurately reflects standard Tachelhit phonology. Dialectal variations or non-standard spelling choices present in the raw Tatoeba lines are reproduced directly into the output script.
• —Loanword Limitations: Foreign names or phrases that lack full-caps formatting or drop outside the explicit manual exception dictionary will undergo native character mapping, occasionally generating unnatural character combinations.