CohereLabs/aya_collection_language_split

**This is a re-upload of the [aya_collection](https://huggingface.co/datasets/CohereLabs/aya_collection), and only differs in the structure of upload. While the original [aya_collection](https://huggingface.co/datasets/CohereLabs/aya_collection) is structured by folders split according to dataset name, this dataset is split by language. We recommend you use this version of the dataset if you are only interested in downloading all of the Aya collection for a single or smaller set of languages.**

Dataset Summary

The Aya Collection is a massive multilingual collection consisting of 513 million instances of prompts and completions covering a wide range of tasks. This collection incorporates instruction-style templates from fluent speakers and applies them to a curated list of datasets, as well as translations of instruction-style datasets into 101 languages. Aya Dataset, a human-curated multilingual instruction and response dataset, is also part of this collection. See our paper for more details regarding the collection.

• —Curated by: Contributors of Aya Open Science Intiative
• —Language(s): 115 languages
• —License: Apache 2.0

• —Aya Datasets Family: | Name | Explanation | |------|--------------| | aya_dataset | Human-annotated multilingual instruction finetuning dataset, comprising over 204K instances across 65 languages. | | aya_collection | Created by applying instruction-style templates from fluent speakers to 44 datasets, including translations of 19 instruction-style datasets into 101 languages. This collection structured based on dataset level subsets. An alternative version of the collection structured by language subsets is also available.| | aya_collection_language_split | Aya Collection structured based on language level subsets. | | aya_evaluation_suite | A diverse evaluation set for multilingual open-ended generation, featuring 250 culturally grounded prompts in 7 languages, 200 translated prompts in 24 languages, and human-edited versions selected for cross-cultural relevance from English Dolly in 6 languages.| | aya_redteaming| A red-teaming dataset consisting of harmful prompts in 8 languages across 9 different categories of harm with explicit labels for "global" and "local" harm.|

Dataset

The Aya Collection is a comprehensive, large corpus of datasets that can be used by researchers around the world to train multilingual models. Our goal is only to include datasets with permissive licensing for manipulation and redistribution.

The Aya Collection consists of three different sources of data:

1. 1.Templated data: We collaborated with fluent speakers to create templates that allowed for the automatic expansion of existing datasets into various languages.
2. 2.Translated data: We translated a hand-selected subset of 19 datasets into 101 languages (114 dialects) using the NLLB 3.3B parameter machine translation model.
3. 3.Aya Dataset: We release the Aya Dataset as a subset of the overall collection. This is the only dataset in the collection that is human-annotated in its entirety.

Load with Datasets

To load this dataset with Datasets, you'll need to install Datasets as pip install datasets --upgrade and then use the following code:

from datasets import load_dataset

dataset = load_dataset("CohereLabs/aya_collection_language_split", "english")

In the above code snippet, "english" refers to a subset of the aya_collection. You can load other subsets by specifying its name at the time of loading the dataset.

Data Instances

An example of a train instance looks as follows:

{'id': 246001,
 'inputs': 'The following query in English is taken from the geography category. What could be the answer to the question?\nWhat is the seventh tallest mountain in North America?',
 'targets': 'The answer is Mount Lucania.',
 'dataset_name': 'Mintaka-inst',
 'sub_dataset_name': '-',
 'task_type': 'question-answering',
 'template_id': 3,
 'language': 'eng',
 'split': 'train',
 'script': 'Latn'
}

Data Fields

The data fields are the same among all splits:

• —id: Unique id of the data point
• —inputs: Prompt or input to the language model.
• —targets: Completion or output of the language model.
• —dataset_name: The name of the source dataset that the data point was taken from
• —sub_dataset_name: If the source is a collection, this field indicates which part of that collection the data point was taken from. If it is not a collection, this field is left blank.
• —task_type: The task type that this conversation belongs to.
• —template_id: The id of the template applied to this data point.
• —language: The ISO code of the dialect of the conversation.
• —script: The script of the language.
• —split: Indicates whether the data point is part of the train or the test split.

Statistics

The total number of data points, including the Aya Dataset` is 513,758,189. To view the breakdown of dialect codes and the respective templated and translated data point counts in the Aya Collection , refer to the toggled table below.

<details> <summary> <b> Breakdown of Aya Collection data point counts grouped by dialects </b> </summary>

PS: Templated data also includes Mozambican Portuguese, which doesn't have its own ISO language code.

</details>

<br>

Motivations & Intentions

• —Curation Rationale: Automatic augmentation of existing datasets serves to enhance the available linguistic resources for multiple languages. The list of languages was initially established from mT5 and aligned with the annotators’ language list and NLLB translation model. The datasets were translated directly from English for all languages.

Additional Information

Provenance

• —Methods Used: A combination of crowd-sourced templating and automatic translation was employed to source this dataset.
• —Methodology Details:
• —Source: Existing NLP datasets
• —Dates of Collection: May 2023 - Dec 2023

Dataset Version and Maintenance

• —Maintenance Status: Actively Maintained
• —Version Details:
• —Current version: 1.0
• —Last Update: 02/2024
• —First Release: 02/2024

Authorship

• —Publishing Organization: Cohere Labs
• —Industry Type: Not-for-profit - Tech
• —Contact Details: https://cohere.com/research/aya

Licensing Information

This dataset can be used for any purpose, whether academic or commercial, under the terms of the Apache 2.0 License.

Citation Information

@misc{singh2024aya,
      title={Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning}, 
      author={Shivalika Singh and Freddie Vargus and Daniel Dsouza and Börje F. Karlsson and Abinaya Mahendiran and Wei-Yin Ko and Herumb Shandilya and Jay Patel and Deividas Mataciunas and Laura OMahony and Mike Zhang and Ramith Hettiarachchi and Joseph Wilson and Marina Machado and Luisa Souza Moura and Dominik Krzemiński and Hakimeh Fadaei and Irem Ergün and Ifeoma Okoh and Aisha Alaagib and Oshan Mudannayake and Zaid Alyafeai and Vu Minh Chien and Sebastian Ruder and Surya Guthikonda and Emad A. Alghamdi and Sebastian Gehrmann and Niklas Muennighoff and Max Bartolo and Julia Kreutzer and Ahmet Üstün and Marzieh Fadaee and Sara Hooker},
      year={2024},
      eprint={2402.06619},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}