cyan12343/frames-benchmark

FRAMES: Factuality, Retrieval, And reasoning MEasurement Set

FRAMES is a comprehensive evaluation dataset designed to test the capabilities of Retrieval-Augmented Generation (RAG) systems across factuality, retrieval accuracy, and reasoning. Our paper with details and experiments is available on arXiv: https://arxiv.org/abs/2409.12941.

Dataset Overview

• —824 challenging multi-hop questions requiring information from 2-15 Wikipedia articles
• —Questions span diverse topics including history, sports, science, animals, health, etc.
• —Each question is labeled with reasoning types: numerical, tabular, multiple constraints, temporal, and post-processing
• —Gold answers and relevant Wikipedia articles provided for each question

Key Features

• —Tests end-to-end RAG capabilities in a unified framework
• —Requires integration of information from multiple sources
• —Incorporates complex reasoning and temporal disambiguation
• —Designed to be challenging for state-of-the-art language models

Usage

This dataset can be used to:

• —Evaluate RAG system performance
• —Benchmark language model factuality and reasoning
• —Develop and test multi-hop retrieval strategies

Baseline Results

We provide baseline results using state-of-the-art models like Gemini-Pro-1.5-0514:

• —Naive prompting: 40.8% accuracy
• —BM25 retrieval (4 docs): 47.4% accuracy
• —Oracle retrieval: 72.9% accuracy
• —Multi-step retrieval & reasoning: 66% accuracy

Citation

If you use this dataset in your research, please cite our paper:

@misc{krishna2024factfetchreasonunified,
      title={Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation}, 
      author={Satyapriya Krishna and Kalpesh Krishna and Anhad Mohananey and Steven Schwarcz and Adam Stambler and Shyam Upadhyay and Manaal Faruqui},
      year={2024},
      eprint={2409.12941},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2409.12941}, 
}

We hope FRAMES will be useful for advancing RAG systems and language model capabilities. For more details, please refer to our full paper.