CoolFace
Datasetpublic

by254126/amnesty_qa

Amnesty QA Dataset A grounded question-answering dataset for evaluating RAG (Retrieval-Augmented Generation) systems, created from reports collected from Amnesty International. This dataset is designed for testing and evaluating RAG pipelines with real-world human rights content. Dataset Structure Each sample contains: user_input: The question to be answered reference: Ground truth answer for evaluation response: Generated answer from the system… See the full description on the dataset page: https://huggingface.co/datasets/by254126/amnesty_qa.

sourceHugging Faceapache-2.0updated 6mo agoView on Hugging Face
0likes2.7kdownloads
Dataset Card

Amnesty QA Dataset

A grounded question-answering dataset for evaluating RAG (Retrieval-Augmented Generation) systems, created from reports collected from Amnesty International.

This dataset is designed for testing and evaluating RAG pipelines with real-world human rights content.

Dataset Structure

Each sample contains:

  • —user_input: The question to be answered
  • —reference: Ground truth answer for evaluation
  • —response: Generated answer from the system
  • —retrieved_contexts: List of relevant context passages retrieved for answering the question

Example

python
{
  'user_input': 'Which private companies in the Americas are the largest GHG emitters according to the Carbon Majors database?',
  'reference': 'The largest private companies in the Americas that are the largest GHG emitters according to the Carbon Majors database are ExxonMobil, Chevron, and Peabody.',
  'response': 'According to the Carbon Majors database, the largest private companies in the Americas that are the largest GHG emitters are:\n\n1. Chevron Corporation (United States)\n2. ExxonMobil Corporation (United States)\n3. ConocoPhillips Company (United States)...',
  'retrieved_contexts': ['The private companies responsible for the most emissions during this period, according to the database, are from the United States: ExxonMobil, Chevron and Peabody.\nThe largest emitter amongst state-owned companies in the Americas is Mexican company Pemex, followed by Venezuelan company Petróleos de Venezuela, S.A.']
}

Usage

python
from datasets import load_dataset

# Load the default English dataset (recommended)
dataset = load_dataset("explodinggradients/amnesty_qa", "english_v3")

# Access the evaluation split
eval_data = dataset["eval"]

# Iterate through samples
for sample in eval_data:
    question = sample["user_input"]
    reference = sample["reference"]
    contexts = sample["retrieved_contexts"]
    # Your RAG evaluation code here

Available Languages

The dataset is available in three languages (all use the v3 schema):

  • —English (recommended): english_v3
  • —Hindi: hindi_v3
  • —Malayalam: malayalam_v3
python
# Load Hindi dataset
dataset = load_dataset("explodinggradients/amnesty_qa", "hindi_v3")

# Load Malayalam dataset
dataset = load_dataset("explodinggradients/amnesty_qa", "malayalam_v3")

Dataset Splits

Only the eval split is available for this dataset, containing 20 carefully curated question-answer pairs.

Legacy Versions

⚠️ Note: Versions v1 and v2 are deprecated and maintained only for backwards compatibility. Please use v3 for all new projects.

<details> <summary>Legacy version schemas (click to expand)</summary>

  • —v1 (deprecated): question, ground_truths (list), answer, contexts
  • —v2 (deprecated): question, ground_truth (string), answer, contexts

</details>

Citation

If you use this dataset in your research, please cite:

bibtex
@dataset{amnesty_qa,
  title = {Amnesty QA: A RAG Evaluation Dataset},
  author = {Exploding Gradients},
  year = {2024},
  publisher = {Hugging Face},
  url = {https://huggingface.co/datasets/explodinggradients/amnesty_qa}
}

License

Apache 2.0