Datasetpublic

Quivr/CRAG

Datasets are taken from Facebook's CRAG: Comprehensive RAG Benchmark, see their arXiv paper for details about the dataset construction. CRAG Sampler We have added a simple Python tool for performing stratified sampling on CRAG data. Installation Local Development Install (Recommended) git clone https://huggingface.co/Quivr/CRAG.git cd CRAG pip install -r requirements.txt # Install dependencies pip install -e . # Install package in development mode… See the full description on the dataset page: https://huggingface.co/datasets/Quivr/CRAG.

sourceHugging Facecc-by-nc-4.0updated 2y agoView on Hugging Face

8likes8.6kdownloads

Dataset Card

Datasets are taken from Facebook's CRAG: Comprehensive RAG Benchmark, see their arXiv paper for details about the dataset construction.

CRAG Sampler

We have added a simple Python tool for performing stratified sampling on CRAG data.

Installation

Local Development Install (Recommended)

bash

git clone https://huggingface.co/Quivr/CRAG.git
cd CRAG
pip install -r requirements.txt  # Install dependencies
pip install -e .  # Install package in development mode

Quick Start

Running the example

bash

python -m examples.basic_sampling

CRAG dataset

CRAG (Comprehensive RAG Benchmark) is a rich and comprehensive factual question answering benchmark designed to advance research in RAG. The public version of the dataset includes:

—2706 Question-Answer pairs
—5 domains: Finance, Sports, Music, Movie, and Open domain
—8 types of questions (see image below): simple, simple with condition, set, comparison, aggregation, multi-hop, post-processing heavy, and false premise

[image]

The datasets crag_task_1_and_2_dev_v4_subsample_*.json.bz2 have been created from the dataset crag_task_1_and_2_dev_v4.jsonl.bz2 available on CRAG's GitHub repository. For an easier handling and download of the dataset, we have used our CRAG sampler to split the 2706 rows of the original file in 5 subsamples, following the procedure below:

1.We have created a new label answer_type, classifying the answers in 3 categories:
2.invalid for any answer == "invalid question"
3.no_answer for any answer == "i don't know"
4.valid for any other answer
5.We have considered the labels answer_type, domain, question_type and static_or_dynamic and performed stratified sampling, splitting the datasets in 5 subsamples. Each subsample has thus the same statistical properties of the full dataset.

We report below the data schema as provided in CRAG's GitHub repository.

Data Schema

Field Name	Type	Description
`interaction_id`	string	A unique identifier for each example.
`query_time`	string	Date and time when the query and the web search occurred.
`domain`	string	Domain label for the query. Possible values: "finance", "music", "movie", "sports", "open". "Open" includes any factual queries not among the previous four domains.
`question_type`	string	Type label about the query. Possible values include: "simple", "simplewcondition", "comparison", "aggregation", "set", "false_premise", "post-processing", "multi-hop".
`static_or_dynamic`	string	Indicates whether the answer to a question changes and the expected rate of change. Possible values: "static", "slow-changing", "fast-changing", and "real-time".
`query`	string	The question for RAG to answer.
`answer`	string	The gold standard answer to the question.
`alt_ans`	list	Other valid gold standard answers to the question.
`split`	integer	Data split indicator, where 0 is for validation and 1 is for the public test.
`search_results`	list of JSON	Contains up to `k` HTML pages for each query (`k=5` for Task #1 and `k=50` for Task #3), including page name, URL, snippet, full HTML, and last modified time.

Search Results Detail

Key	Type	Description
`page_name`	string	The name of the webpage.
`page_url`	string	The URL of the webpage.
`page_snippet`	string	A short paragraph describing the major content of the page.
`page_result`	string	The full HTML of the webpage.
`page_last_modified`	string	The time when the page was last modified.