abdelhaqueidali/Kabyle-Latin-to-Tifinagh-Parallel-Corpus

Dataset Card for Kabyle Latin-to-Tifinagh Parallel Corpus

This dataset provides a parallel corpus of the Kabyle language (Taqbaylit), pairing native Latin-based orthography with automated, context-aware Amazigh script transliterations. It is built by processing raw text data through a rule-based algorithmic pipeline designed to enforce strict orthographic purity, manage contextual phonetic mutations, and isolate foreign vocabulary.

Dataset Details

Dataset Description

• —Curated by:
• —Language(s) (NLP): Kabyle (Latin script and Amazigh script)
• —License: CC-BY 2.0 (Inherited from the Tatoeba source corpus)

Dataset Sources

• —Repository: [More Information Needed]
• —Source Project: Tatoeba Translation Corpus

Uses

Direct Use

• —Training and fine-tuning machine translation models and sequence-to-sequence tokenizers.
• —Evaluating local-first or low-resource script normalizers and converters.

Out-of-Scope Use

This dataset is strictly optimized for native phonetic structures. Text processing forcefully filters out unadapted loanwords, making it unsuited for generalized code-switching (e.g., raw French-Kabyle mixed text) that does not adhere to standard Amazigh structural frameworks.

Dataset Structure

The dataset contains three distinct configuration subsets formatted as Tab-Separated Values (.tsv) files with specific schemas:

1. Monolingual Subset (kab_tifinagh)

• —id: Unique identifier inherited from the Tatoeba database.
• —kab_tfng: The processed Amazigh alphabet transliteration column.
• —kab_latn: The original source Kabyle sentence in standard Latin script.

2. English Parallel Subset (en_kab_tifinagh)

• —id: Unique sentence identifier.
• —kab_tfng: The processed Amazigh alphabet transliteration column.
• —kab_latn: The source Kabyle sentence in standard Latin script.
• —en: Corresponding English translation text string.

3. French Parallel Subset (fr_kab_tifinagh)

• —id: Unique sentence identifier.
• —kab_tfng: The processed Amazigh alphabet transliteration column.
• —kab_latn: The source Kabyle sentence in standard Latin script.
• —fr: Corresponding French translation text string.

Dataset Creation

Curation Rationale

While the Kabyle language is widely written using a modified Latin alphabet, the standardized Amazigh alphabet holds deep official and cultural significance. This dataset bridges the script divide algorithmically, applying rigorous phonetic harmony and morphological adjustments to build a high-fidelity parallel repository.

Source Data

Data Collection and Processing

The source texts are extracted from the Kabyle subsets of the Tatoeba project. The pipeline maps characters directly to the 33 official IRCAM standardized alphabet characters while executing a multi-tier linguistic and structural engine:

1. Character Map & Realignment Graphemes

The pipeline uses a strict character matrix reducing entries directly to the core script block, explicitly accommodating gemination, labialization, and target replacements:

• —Geminated Affricates: čč => ⵜⵜⵛ, ǧǧ => ⴷⴷⵊ, žž => ⴷⴷⵣ, ţţ => ⵜⵜ.
• —Labialization Graphemes: gʷ => ⴳⵯ, kʷ => ⴽⵯ.
• —Standard Alternations: č => ⵜⵛ, ǧ => ⴷⵊ, ž => ⴷⵣ, ţ => ⵜⵜ.
• —Foreign Borrowing Adaptations: Maps validated morphological structures containing p => ⴱ and v => ⴼ.

2. Grammatical Dash Merging

Structural hyphens used for direct/indirect objects, prepositions, or kinship suffixes are automatically stripped, merging dependent particles cleanly into single word tokens:

• —Prepositional/Object Frameworks: fell-as => ⴼⴻⵍⵍⴰⵙ, ger-aɣ => ⴳⴻⵔⴰⵖ.
• —Locative/Locational Frameworks: ɣur-sen => ⵖⵓⵔⵙⴻⵏ, deg-sen => ⴷⴻⴳⵙⴻⵏ.
• —Kinship Expressions: baba-s => ⴱⴰⴱⴰⵙ, mmi-k => ⵎⵎⵉⴽ.
• —Note: Non-grammatical structural dashes are globally normalized to empty spaces.

3. Contextual Phonetics & Schwa (ⴻ) Optimization

• —Initial Mutations: Word-initial sequences starting with ye or we migrate to native vocalic variants i and u respectively.
• —Vocalic Semivowel Linearizations: Sequences matching ew or ey followed by a native consonant or sitting at the absolute end of a word are flattened directly into u or i.
• —Pharyngealization Harmony: Progressive and regressive assimilation constraints force the plain alveolar trill r / R to upgrade into its pharyngealized counterpart ṛ / Ṛ if it directly borders or is separated by a single vowel from any pharyngeal anchor (ẓ, ḍ, ṭ, ṣ).
• —Schwa Stripping: The engine drops an initial schwa (ⴻ) at the absolute start of a word. Internally, ⴻ is dynamically dropped between two distinct consonants using a strict regex lookahead filter, but remains preserved when sandwiched between identical, geminated consonants.

4. Strict Isolation Filter (Skipped Sentences)

To prevent phonetic distortion, a row is completely discarded and sent to a _skipped.tsv ledger if a token meets any of the following conditions:

• —Contains French accented characters (é, è, â, ç, etc.).
• —Contains true diphthongs (doubled vowel sequences like ae, ui, oi).
• —Contains the vowel o or O unless it represents a structural character assignment or explicit native override.
• —Terminates explicitly with an unmapped trailing e or E.
• —Contains an apostrophe (') or represents an all-caps acronym (e.g., MRI).
• —Contains standalone capital letters internally or at the tail end of strings.
• —Fails a final structural purity regex check (^[0-9ⵯ\u2D30-\u2D6F\s.,!?;:()""\']+$).

5. Target Overrides

High-frequency cross-linguistic tokens are explicitly caught by a manual override filter before any exclusion rules trigger:

• —Tom / ṭom / ṭum => ⵟⵓⵎ
• —Ait / ait => ⴰⵢⵜ
• —Mary => ⵎⴰⵔⵉ

Who are the source data producers?

The underlying source corpus sentences are crowdsourced, verified, and translated by independent contributors and native speakers on the open-source Tatoeba platform.

Bias, Risks, and Limitations

Because the translation relies heavily on an automated algorithmic pipeline to catch external loanword structures, structural edge cases can occur:

• —Loanwords that naturally lack explicit French accented markers, trailing es, or common vowels like o may slip past the isolation filters and undergo incorrect, literal transliteration into the Amazigh script.
• —Rare native structures or highly localized dialectal expressions matching an exclusion rule rule profile might be inadvertently skipped.