CohereLabs/aya_evaluation_suite

Dataset Summary

Aya Evaluation Suite contains a total of 26,750 open-ended conversation-style prompts to evaluate multilingual open-ended generation quality.\ To strike a balance between language coverage and the quality that comes with human curation, we create an evaluation suite that includes: 1) human-curated examples in 7 languages (tur, eng, yor, arb, zho, por, tel) → aya-human-annotated. 2) machine-translations of handpicked examples into 101 languages → dolly-machine-translated. 3) human-post-edited translations into 6 languages (hin, srp, rus, fra, arb, spa) → dolly-human-edited.

• —Curated by: Contributors of Aya Open Science Intiative, professional annotators, and synthetic generation
• —Language(s): 101 languages
• —License: Apache 2.0
• —Aya Datasets Family: | Name | Explanation | |------|--------------| | aya_dataset | Human-annotated multilingual instruction finetuning dataset, comprising over 204K instances across 65 languages. | | aya_collection | Created by applying instruction-style templates from fluent speakers to 44 datasets, including translations of 19 instruction-style datasets into 101 languages, providing 513M instances for various tasks.| | aya_collection_language_split | Aya Collection structured based on language level subsets. | | aya_evaluation_suite | A diverse evaluation set for multilingual open-ended generation, featuring 250 culturally grounded prompts in 7 languages, 200 translated prompts in 24 languages, and human-edited versions selected for cross-cultural relevance from English Dolly in 6 languages.| | aya_redteaming| A red-teaming dataset consisting of harmful prompts in 8 languages across 9 different categories of harm with explicit labels for "global" and "local" harm.|

Dataset

The Aya Evaluation Suite includes the following subsets:

1. 1.aya-human-annotated: 250 original human-written prompts in 7 languages each.
2. 2.dolly-machine-translated: 200 human-selected prompts from databricks-dolly-15k , automatically translated with the NLLB model from English into 101 languages (114 dialects in total).
3. 3.dolly-human-edited: 200 dolly-machine-translated prompts post-edited by fluent speakers for 6 languages.

Load with Datasets

To load this dataset consisting of prompt-completions with datasets, you just need to install Datasets as pip install datasets --upgrade and then use the following code:

from datasets import load_dataset

aya_eval = load_dataset("CohereLabs/aya_evaluation_suite", "aya_human_annotated")

Data Fields

• —id: Unique id of the data point.
• —inputs: Prompt or input to the language model.
• —targets: Completion or output of the language model. (Not applicable for dolly-human-edited)
• —language: The language of the prompt and completion.
• —script: The writing system of the language.
• —source_id: Corresponding original row index from the databricks-dolly-15k dataset (Field applicable only for subsets dolly-machine-translated & dolly-human-edited)

Data Instances

Example data instances from the Aya Evaluation Suite subsets are listed in the toggled sections below.

<details> <summary> <b>aya-human-annotated</b> </summary>

{
"id": 42,
"inputs": "What day is known as Star Wars Day?",
"targets": "May 4th (May the 4th be with you!)",
"language": "eng",
"script": "Latn",
}

</details>

<b>Dolly-machine-translated and dolly-human-edited</b>

• —These two subsets are parallel datasets (data instances can be mapped using their id column).
• —Note that in the dolly-machine-translated subset, we also include the original English subset (id 1-200), which is translated into 101 languages. Furthermore, the field id can be used to match the translations of the same data instance across languages.
• —The source_id field contains the corresponding original row index from the databricks-dolly-15k dataset. <details> <summary> <b>dolly-machine-translated</b> </summary>

  {
  "id": 2,
  "inputs": "How to escape from a helicopter trapped in water ?",
  "targets": "If you are ever trapped inside a helicopter while submerged in water, it’s best to try and remain calm until the cabin is completely underwater. It’s better to wait for pressure to be equalized, before you try to open the door or break the glass to escape.",
  "language": "eng",
  "script": "Latn",
  "source_id": 6060,
  }

</details>

<details> <summary> <b>dolly-human-edited</b> </summary>

  {
  "id": 2,
  "inputs": "Comment peut-on s'échapper d'un hélicoptère piégé dans l'eau ?",
  "targets": "-",
  "language": "fra",
  "script": "Latn",
  "source_id": 6060,
  }

</details>

Statistics

The toggled table below lists the breakdown of languages in each subset.

Languages

<details> <summary> <b>aya-human-annotated</b> </summary>

</details>

<details> <summary> <b>dolly-machine-translated</b> </summary>

</details>

<details> <summary> <b>dolly-human-edited</b> </summary>

</details>

<br>

Motivations & Intentions

• —Curation Rationale: This evaluation suite is tailored to test the generation quality of multilingual models, with the aim of balancing language coverage and human-sourced quality. It covers prompts originally written in each language, as well as English-centric translated, and manually curated or edited prompts for a linguistically broad, but rich testbed. The list of languages was initially established from mT5 and aligned with the annotators’ language list and the NLLB translation model.

Known Limitations

• —Translation Quality: Note that the expressiveness of the dolly-machine-translated subset is limited by the quality of the translation model and may adversely impact an estimate of ability in languages where translations are not adequate. If this subset is used for testing, we recommend it be paired and reported with the professionally post-edited dolly-human-edited subset or the aya-human-annotated set, which, while covering only 7 languages, is entirely created by proficient target language speakers. ---

Additional Information

Provenance

• —Methods Used: combination of original annotations by volunteers, automatic translation, and post-editing of translations by professional annotators.
• —Methodology Details:
• —Source: Original annotations from Aya dataset along with translations and post-edits of Dolly dataset
• —Platform: Aya Annotation Platform
• —Dates of Collection: May 2023 - Dec 2023

Dataset Version and Maintenance

• —Maintenance Status: Actively Maintained
• —Version Details:
• —Current version: 1.0
• —Last Update: 02/2024
• —First Release: 02/2024
• —Maintenance Plan: No updates planned.

Authorship

• —Publishing Organization: Cohere Labs
• —Industry Type: Not-for-profit - Tech
• —Contact Details: https://aya.for.ai/

Licensing Information

This dataset can be used for any purpose, whether academic or commercial, under the terms of the Apache 2.0 License.

Citation Information

@misc{singh2024aya,
      title={Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning}, 
      author={Shivalika Singh and Freddie Vargus and Daniel Dsouza and Börje F. Karlsson and Abinaya Mahendiran and Wei-Yin Ko and Herumb Shandilya and Jay Patel and Deividas Mataciunas and Laura OMahony and Mike Zhang and Ramith Hettiarachchi and Joseph Wilson and Marina Machado and Luisa Souza Moura and Dominik Krzemiński and Hakimeh Fadaei and Irem Ergün and Ifeoma Okoh and Aisha Alaagib and Oshan Mudannayake and Zaid Alyafeai and Vu Minh Chien and Sebastian Ruder and Surya Guthikonda and Emad A. Alghamdi and Sebastian Gehrmann and Niklas Muennighoff and Max Bartolo and Julia Kreutzer and Ahmet Üstün and Marzieh Fadaee and Sara Hooker},
      year={2024},
      eprint={2402.06619},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}