google/frames-benchmark

1---2license: apache-2.03language:4- en5tags:6- rag7- long-context8- llm-search9- reasoning10- factuality11- retrieval12- question-answering13- iterative-search14task_categories:15- text-classification16- token-classification17- table-question-answering18- question-answering19pretty_name: Who are I or you20size_categories:21- n>1T22---23 24# FRAMES: Factuality, Retrieval, And reasoning MEasurement Set25 26FRAMES is a comprehensive evaluation dataset designed to test the capabilities of Retrieval-Augmented Generation (RAG) systems across factuality, retrieval accuracy, and reasoning.27Our paper with details and experiments is available on arXiv: [https://arxiv.org/abs/2409.12941](https://arxiv.org/abs/2409.12941).28 29 30## Dataset Overview31 32- 824 challenging multi-hop questions requiring information from 2-15 Wikipedia articles33- Questions span diverse topics including history, sports, science, animals, health, etc.34- Each question is labeled with reasoning types: numerical, tabular, multiple constraints, temporal, and post-processing35- Gold answers and relevant Wikipedia articles provided for each question36 37## Key Features38 39- Tests end-to-end RAG capabilities in a unified framework40- Requires integration of information from multiple sources41- Incorporates complex reasoning and temporal disambiguation42- Designed to be challenging for state-of-the-art language models43 44## Usage45 46This dataset can be used to:47- Evaluate RAG system performance 48- Benchmark language model factuality and reasoning49- Develop and test multi-hop retrieval strategies50 51## Baseline Results52 53We provide baseline results using state-of-the-art models like Gemini-Pro-1.5-0514:54 55- Naive prompting: 40.8% accuracy56- BM25 retrieval (4 docs): 47.4% accuracy  57- Oracle retrieval: 72.9% accuracy58- Multi-step retrieval & reasoning: 66% accuracy59 60## Citation61 62If you use this dataset in your research, please cite our paper:63 64```65@misc{krishna2024factfetchreasonunified,66      title={Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation}, 67      author={Satyapriya Krishna and Kalpesh Krishna and Anhad Mohananey and Steven Schwarcz and Adam Stambler and Shyam Upadhyay and Manaal Faruqui},68      year={2024},69      eprint={2409.12941},70      archivePrefix={arXiv},71      primaryClass={cs.CL},72      url={https://arxiv.org/abs/2409.12941}, 73}74```75 76We hope FRAMES will be useful for advancing RAG systems and language model capabilities. For more details, please refer to our full paper.