achermiti/derja

For reference on dataset card metadata, see the spec: https://github.com/huggingface/hub-docs/blob/main/datasetcard.md?plain=1

Doc / guide: https://huggingface.co/docs/hub/datasets-cards

{} ---

Dataset Card for Tunisian Derja to English Translation Dataset

Dataset Details

Dataset Description

This dataset contains approximately 13,036 rows of sentence-level translations from Tunisian Derja (a dialect of Arabic) to English. The dataset includes diverse examples from everyday conversations, colloquial phrases, and idiomatic expressions in Tunisian Derja. The dataset has been curated to facilitate tasks such as machine translation, natural language processing (NLP), and linguistic research specific to Tunisian dialects.

• —Curated by: Khawla Jlassi, Imen Ayari, Jacer Dabbabi
• —Funded by [optional]: No funding information provided.
• —Shared by: Dataset contributors.
• —Language(s) (NLP): Tunisian Derja (ar_TN), English (en).
• —License: MIT License (or select appropriate license).

Dataset Sources

• —Repository: GitHub Repository
• —Paper [optional]: No paper available at the moment.
• —Demo [optional]: Not provided.

Uses

Direct Use

This dataset can be directly used to fine-tune machine translation models, especially for translating Tunisian Derja to English. It is suited for applications in translation, language modeling, linguistic research, and creating language learning tools for non-native speakers.

Out-of-Scope Use

The dataset is not suitable for formal Arabic to English translations or tasks requiring Modern Standard Arabic (MSA) as it only contains informal, colloquial Tunisian Derja phrases. It is not recommended for sensitive contexts where precise legal, medical, or formal language translation is required.

Dataset Structure

• —Input Column (Derja): Contains sentences in Tunisian Derja.
• —Output Column (English): Contains translations of these sentences in English.
• —The dataset includes approximately 13,036 sentences, split into training and validation sets, for NLP tasks like machine translation.

Dataset Creation

Curation Rationale

This dataset was created to bridge the gap in available resources for machine translation between Tunisian Derja and English. Tunisian Derja is an under-resourced dialect of Arabic, and this dataset enables researchers and developers to build tools for translation, language learning, and dialect recognition.

Source Data

Data Collection and Processing

The dataset was sourced from various informal contexts, including social media, colloquial speech, and everyday conversations. The sentences were scraped from a public source (Derja Ninja) and cleaned to ensure proper sentence formatting. The dataset was processed and split into training and validation sets for machine learning tasks.

Who are the source data producers?

The original data producers are native speakers of Tunisian Derja who contributed to the dataset via public conversations and social media posts. The sentences were selected based on relevance to common expressions and conversational usage in Tunisia.

Annotations

Annotation Process

The dataset does not include manual annotations beyond the translations from Tunisian Derja to English. No specific annotation tools were used, as the focus is on machine translation tasks.

Who are the annotators?

The annotators are contributors to the GitHub project, who are proficient in both Tunisian Derja and English.

Personal and Sensitive Information

No personal, sensitive, or private information is included in the dataset. The sentences are anonymized and consist solely of everyday, non-identifiable phrases.

Bias, Risks, and Limitations

The dataset may contain biases based on the informal nature of Tunisian Derja, which varies widely between regions in Tunisia. Some dialect variations may not be well-represented, and translations may not capture the exact meaning of certain idiomatic expressions. Users should exercise caution when applying the dataset for applications requiring precise translations.

Recommendations

It is recommended to use the dataset primarily for educational, research, and experimental translation tasks. Users should be aware of dialectal differences and potential cultural context in the translations.

Citation

BibTeX:

@misc{jlassi1_2024,
  title={Tunisian Derja to English Translation Dataset},
  author={Khawla Jlassi, Imen Ayari, Jacer Dabbabi},
  year={2024},
  url={https://github.com/jlassi1/Machine_Translation_Derjja_English},
}