Datasetpublic

Akjhtar/raid

🚨 RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors 🚨 🌐 Website, 🖥️ Github, 📝 Paper RAID is the largest & most comprehensive dataset for evaluating AI-generated text detectors. It contains over 10 million documents spanning 11 LLMs, 11 genres, 4 decoding strategies, and 12 adversarial attacks. It is designed to be the go-to location for trustworthy third-party evaluation of both open-source and closed-source generated text detectors. Load… See the full description on the dataset page: https://huggingface.co/datasets/Akjhtar/raid.

sourceHugging Facemitupdated 1mo agoView on Hugging Face

0likes60downloads

Dataset Card

<div align="center"> <h1>🚨 RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors 🚨</h1> 🌐 <a href="https://raid-bench.xyz"><b>Website</b></a>, 🖥️ <a href="https://github.com/liamdugan/raid"><b>Github</b></a>, 📝 <a href="https://arxiv.org/abs/2405.07940"><b>Paper</b></a> </div>

RAID is the largest & most comprehensive dataset for evaluating AI-generated text detectors. It contains over 10 million documents spanning 11 LLMs, 11 genres, 4 decoding strategies, and 12 adversarial attacks. It is designed to be the go-to location for trustworthy third-party evaluation of both open-source and closed-source generated text detectors.

Load the dataset

To load the dataset, install the library datasets with pip install datasets. Then,

from datasets import load_dataset
raid = load_dataset("liamdugan/raid")

Data fields

The RAID dataset has the following fields

1. `id`: A uuid4 that uniquely identifies the original content of the generation
2. `adv_source_id`: uuid4 of the source of the generation if adversarial
3. `source_id`: uuid4 of the human-written source text
4. `model`: The model that generated the text
   - Choices: `['chatgpt', 'gpt4', 'gpt3', 'gpt2', 'llama-chat', 'mistral', 'mistral-chat', 'mpt', 'mpt-chat', 'cohere', 'cohere-chat']`
5. `decoding`: The decoding strategy used 
    - Choices: `['greedy', 'sampling']`
6. `repetition_penalty`: Whether or not we use a repetition penalty of 1.2 when generating
    - Choices: `['yes', 'no']`
7. `attack`: The adversarial attack used
    - Choices: `['homoglyph', 'number', 'article_deletion', 'insert_paragraphs', 'perplexity_misspelling', 'upper_lower', 'whitespace', 'zero_width_space', 'synonym', 'paraphrase', 'alternative_spelling']`
8. `domain`: The genre from where the prompt/text was taken
    - Choices: `['abstracts', 'books', 'code', 'czech', 'german', 'news', 'poetry', 'recipes', 'reddit', 'reviews', 'wiki']`
9. `title`: The title of the article used in the prompt
10. `prompt`: The prompt used to generate the text
11. `generation`: The text of the generation

Data splits

&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;	Labels?	Domains	Dataset Size (w/o adversarial)	Dataset Size (w/ adversarial)
RAID-train	✅	News, Books, Abstracts, Reviews, Reddit, Recipes, Wikipedia, Poetry	802M	11.8G
RAID-test	❌	News, Books, Abstracts, Reviews, Reddit, Recipes, Wikipedia, Poetry	81.0M	1.22G
RAID-extra	✅	Code, Czech, German	275M	3.71G

Citation

BibTeX:

bibtex

@inproceedings{dugan-etal-2024-raid,
    title = "{RAID}: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors",
    author = "Dugan, Liam  and
      Hwang, Alyssa  and
      Trhl{\'\i}k, Filip  and
      Zhu, Andrew  and
      Ludan, Josh Magnus  and
      Xu, Hainiu  and
      Ippolito, Daphne  and
      Callison-Burch, Chris",
    booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = aug,
    year = "2024",
    address = "Bangkok, Thailand",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.acl-long.674",
    pages = "12463--12492",
}