yrrhall/Arabic_Aya

Dataset Card for : Arabic Aya (2A)

<!-- Provide a quick summary of the dataset. -->

<!-- This dataset card aims to be a base template for new datasets. It has been generated using this raw template.-->

Arabic Aya (2A) : A Curated Subset of the Aya Collection for Arabic Language Processing

Dataset Sources & Infos

• —Data Origin: Derived from 69 subsets of the original Aya datasets : CohereForAI/aya_collection, CohereForAI/aya_dataset, and CohereForAI/aya_evaluation_suite.
• —Languages: Modern Standard Arabic (MSA) and a variety of Arabic dialects ( 'arb', 'arz', 'ary', 'ars', 'knc', 'acm', 'apc', 'aeb', 'ajp', 'acq' )
• —Applications: Language Modeling, Text Classification, Sentiment Analysis, Dialect Identification, Translation
• —Paper: 2402.06619
• —Maintainer: Elfilali Ali
• —License: Apache-2.0

Overview

Arabic Aya is a meticulously curated dataset derived from the comprehensive Aya collection by CohereForAI, specifically focusing on Arabic text data. This dataset aggregates content from the CohereForAI/aya_collection, CohereForAI/aya_dataset, and CohereForAI/aya_evaluation_suite, filtering out all but the Arabic content, including both Modern Standard Arabic (MSA) and various regional dialects.

Purpose

The aim of 'Arabic Aya' is to provide researchers, technologists, and linguists with a ready-to-use Arabic text resource, significantly reducing the time and effort required for data preprocessing in NLP and AI projects focused on the Arabic language.

• —Use the Aya datasets out of the box for your Arabic applications and research 😀

Usage

This dataset serves as a foundational tool for those embarking on Arabic language projects, from academic research to commercial applications. By providing a pre-filtered source of Arabic text, 'Arabic Aya' enables users to dive straight into model training, analysis, and application development without the preliminary hassle of data cleaning and language filtering.

Use with HuggingFace's datasets library

To load this dataset with Datasets, you'll need to install Datasets as pip install datasets --upgrade and then use a similar code to the following:

from datasets import load_dataset

dataset = load_dataset("2A2I/Arabic_Aya", "CohereForAI-aya_collection-templated_mintaka")

In the above code snippet, "CohereForAI-ayacollection-templatedmintaka" refers to the arabic version (100k rows) of the original "templatedmintaka" subset (780k rows) of the ayacollection. You can load other subsets by specifying its name at the time of loading the dataset.

Access and Contribution

Available on the Hugging Face Hub under 2A2I/Arabic_Aya, 'Arabic Aya' invites contributions from the community. Users are encouraged to offer feedback, suggest improvements.

Support and Collaboration

We are committed to fostering an inclusive and supportive environment around Arabic AI and NLP research. For support, collaboration, or queries regarding the dataset, please reach out through the Hugging Face Hub's discussion section or reach out at 2A2I Contact Email.

Original Dataset Card of Aya by CohereForAI

Dataset Summary

The Aya Collection is a massive multilingual collection consisting of 513 million instances of prompts and completions covering a wide range of tasks. This collection incorporates instruction-style templates from fluent speakers and applies them to a curated list of datasets, as well as translations of instruction-style datasets into 101 languages. Aya Dataset, a human-curated multilingual instruction and response dataset, is also part of this collection. See our paper for more details regarding the collection.

• —Curated by: Contributors of Aya Open Science Intiative
• —Language(s): 115 languages
• —License: Apache 2.0

• —Aya Datasets Family: | Name | Explanation | |------|--------------| | aya_dataset | Human-annotated multilingual instruction finetuning dataset, comprising over 204K instances across 65 languages. | | aya_collection | Created by applying instruction-style templates from fluent speakers to 44 datasets, including translations of 19 instruction-style datasets into 101 languages.| | aya_evaluation_suite | A diverse evaluation set for multilingual open-ended generation, featuring 250 culturally grounded prompts in 7 languages, 200 translated prompts in 24 languages, and human-edited versions selected for cross-cultural relevance from English Dolly in 6 languages.|

Dataset

The Aya Collection is a comprehensive, large corpus of datasets that can be used by researchers around the world to train multilingual models. Our goal is only to include datasets with permissive licensing for manipulation and redistribution.

The Aya Collection consists of three different sources of data:

1. 1.Templated data: We collaborated with fluent speakers to create templates that allowed for the automatic expansion of existing datasets into various languages.
2. 2.Translated data: We translated a hand-selected subset of 19 datasets into 101 languages (114 dialects) using the NLLB 3.3B parameter machine translation model.
3. 3.Aya Dataset: We release the Aya Dataset as a subset of the overall collection. This is the only dataset in the collection that is human-annotated in its entirety.

Load with Datasets

To load this dataset with Datasets, you'll need to install Datasets as pip install datasets --upgrade and then use the following code:

from datasets import load_dataset

dataset = load_dataset("CohereForAI/aya_collection", "templated_mintaka")

In the above code snippet, "templatedmintaka" refers to a subset of the ayacollection. You can load other subsets by specifying its name at the time of loading the dataset.

Data Instances

An example of a train instance looks as follows:

{'id': 246001,
 'inputs': 'The following query in English is taken from the geography category. What could be the answer to the question?\nWhat is the seventh tallest mountain in North America?',
 'targets': 'The answer is Mount Lucania.',
 'dataset_name': 'Mintaka-inst',
 'sub_dataset_name': '-',
 'task_type': 'question-answering',
 'template_id': 3,
 'language': 'eng',
 'split': 'train',
 'script': 'Latn'
}

Data Fields

The data fields are the same among all splits:

• —id: Unique id of the data point
• —inputs: Prompt or input to the language model.
• —targets: Completion or output of the language model.
• —dataset_name: The name of the source dataset that the data point was taken from
• —sub_dataset_name: If the source is a collection, this field indicates which part of that collection the data point was taken from. If it is not a collection, this field is left blank.
• —task_type: The task type that this conversation belongs to.
• —template_id: The id of the template applied to this data point.
• —language: The ISO code of the dialect of the conversation.
• —script: The script of the language.
• —split: Indicates whether the data point is part of the train or the test split.

Statistics

The total number of data points, including the Aya Dataset` is 513,758,189. To view the breakdown of dialect codes and the respective templated and translated data point counts in the Aya Collection , refer to the toggled table below.

<details> <summary> <b> Breakdown of Aya Collection data point counts grouped by dialects </b> </summary>

PS: Templated data also includes Mozambican Portuguese, which doesn't have its own ISO language code.

</details>

<br>

Motivations & Intentions

• —Curation Rationale: Automatic augmentation of existing datasets serves to enhance the available linguistic resources for multiple languages. The list of languages was initially established from mT5 and aligned with the annotators’ language list and NLLB translation model. The datasets were translated directly from English for all languages.

Additional Information

Provenance

• —Methods Used: A combination of crowd-sourced templating and automatic translation was employed to source this dataset.
• —Methodology Details:
• —Source: Existing NLP datasets
• —Dates of Collection: May 2023 - Dec 2023

Dataset Version and Maintenance

• —Maintenance Status: Actively Maintained
• —Version Details:
• —Current version: 1.0
• —Last Update: 02/2024
• —First Release: 02/2024

Authorship

• —Publishing Organization: Cohere For AI
• —Industry Type: Not-for-profit - Tech
• —Contact Details: https://cohere.com/research/aya

Licensing Information

This dataset can be used for any purpose, whether academic or commercial, under the terms of the Apache 2.0 License.

Citation Information

@misc{singh2024aya,
      title={Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning}, 
      author={Shivalika Singh and Freddie Vargus and Daniel Dsouza and Börje F. Karlsson and Abinaya Mahendiran and Wei-Yin Ko and Herumb Shandilya and Jay Patel and Deividas Mataciunas and Laura OMahony and Mike Zhang and Ramith Hettiarachchi and Joseph Wilson and Marina Machado and Luisa Souza Moura and Dominik Krzemiński and Hakimeh Fadaei and Irem Ergün and Ifeoma Okoh and Aisha Alaagib and Oshan Mudannayake and Zaid Alyafeai and Vu Minh Chien and Sebastian Ruder and Surya Guthikonda and Emad A. Alghamdi and Sebastian Gehrmann and Niklas Muennighoff and Max Bartolo and Julia Kreutzer and Ahmet Üstün and Marzieh Fadaee and Sara Hooker},
      year={2024},
      eprint={2402.06619},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}