datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
AdvBench
Dataset Card for AdvBench
Paper: Universal and Transferable Adversarial Attacks on Aligned Language Models
Data: AdvBench Dataset
About
AdvBench is a set of 500 harmful behaviors formulated as instructions. These behaviors
range over the same themes as the harmful strings setting, but the adversary’s goal
is instead to find a single attack string that will cause the model to generate any response
that attempts to comply with the instruction, and to do so over as many… See the full description on the dataset page: https://huggingface.co/datasets/walledai/AdvBench.XSTest
XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models
Paper: XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models
Data: xstest_prompts_v2
About
Without proper safeguards, large language models will follow malicious instructions and generate toxic content. This motivates safety efforts such as red-teaming and large-scale feedback learning, which aim to make models both helpful and harmless.… See the full description on the dataset page: https://huggingface.co/datasets/walledai/XSTest.HarmBench
HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
Paper: HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal
Data: Dataset
About
In this dataset card, we only use the behavior prompts proposed in HarmBench.
License
MIT
Citation
If you find HarmBench useful in your research, please consider citing the paper:
@article{mazeika2024harmbench,
title={HarmBench: A… See the full description on the dataset page: https://huggingface.co/datasets/walledai/HarmBench.JailbreakBench
JailbreakBench: An Open Robustness Benchmark for Jailbreaking Language Models
Paper: JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models
Data: JailbreaBench-HFLink
About
Jailbreakbench is an open-source robustness benchmark for jailbreaking large language models (LLMs). The goal of this benchmark is to comprehensively track progress toward (1) generating successful jailbreaks and (2) defending against these jailbreaks. To this end, we… See the full description on the dataset page: https://huggingface.co/datasets/walledai/JailbreakBench.JailbreakHub
In-The-Wild Jailbreak Prompts on LLMs
Paper: ``Do Anything Now'': Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models
Data: Dataset
Data
Prompts
Overall, authors collect 15,140 prompts from four platforms (Reddit, Discord, websites, and open-source datasets) during Dec 2022 to Dec 2023. Among these prompts, they identify 1,405 jailbreak prompts. To the best of our knowledge, this dataset serves as the largest collection of… See the full description on the dataset page: https://huggingface.co/datasets/walledai/JailbreakHub.StrongREJECT
StrongREJECT
A novel benchmark of 313 malicious prompts for use in evaluating jailbreaking attacks against LLMs, aimed to expose whether a jailbreak attack actually enables malicious actors to utilize LLMs for harmful tasks.
Dataset link: https://github.com/alexandrasouly/strongreject/blob/main/strongreject_dataset/strongreject_dataset.csv
Citation
If you find the dataset useful, please cite the following work:
@misc{souly2024strongreject,
title={A StrongREJECT… See the full description on the dataset page: https://huggingface.co/datasets/walledai/StrongREJECT.MaliciousInstruct
Malicious Instruct
The dataset is obtained from the paper: Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation and is available here in the source repository.
Citation
If you use this dataset, please consider citing the following work:
@article{huang2023catastrophic,
title={Catastrophic jailbreak of open-source llms via exploiting generation},
author={Huang, Yangsibo and Gupta, Samyak and Xia, Mengzhou and Li, Kai and Chen, Danqi},
journal={arXiv… See the full description on the dataset page: https://huggingface.co/datasets/walledai/MaliciousInstruct.BBQ
BBQ
Repository for the Bias Benchmark for QA dataset.
https://github.com/nyu-mll/BBQ
Authors: Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R. Bowman.
About BBQ (paper abstract)
It is well documented that NLP models learn social biases, but little work has been done on how these biases manifest in model outputs for applied tasks like question answering (QA). We introduce the Bias Benchmark for QA… See the full description on the dataset page: https://huggingface.co/datasets/walledai/BBQ.CatHarmfulQA
Dataset Card for CatQA
Paper: Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic
Data: CatQA Dataset
About
CatQA is used in LLM safety realignment research as a categorical harmful questions dataset. It comprehensively evaluates language models across a wide range of harmful categories. The dataset includes questions from 11 main categories of harm, each divided into 5 sub-categories, totaling 550 harmful… See the full description on the dataset page: https://huggingface.co/datasets/walledai/CatHarmfulQA.WildGuardTest
Dataset Card for WildGuardMix
Paper: WildGuard: Open One-stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs
Data: WildGuardMix Dataset
Disclaimer
The data includes examples that might be disturbing, harmful, or upsetting. It covers discriminatory language, discussions about abuse, violence, self-harm, sexual content, misinformation, and other high-risk categories. It is recommended not to train a Language Model exclusively on the harmful examples.… See the full description on the dataset page: https://huggingface.co/datasets/walledai/WildGuardTest.WildJailbreak
WildJailbreak
Paper: WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models
Data: DatasetHF_link
WildJailbreak Dataset Card
WildJailbreak is an open-source synthetic safety-training dataset with 262K vanilla (direct harmful requests) and adversarial (complex adversarial jailbreaks) prompt-response pairs. In order to mitigate exaggerated safety behaviors, WildJailbreaks provides two contrastive types of queries: 1) harmful queries (both… See the full description on the dataset page: https://huggingface.co/datasets/walledai/WildJailbreak.TDC23-RedTeaming
TDC 2023 (LLM Edition) - Red Teaming Track
This is the combined dev and test set from the Red Teaming Track of TDC 2023.
Citation
If find this dataset useful, please cite the following work:
@inproceedings{tdc2023,
title={TDC 2023 (LLM Edition): The Trojan Detection Challenge},
author={Mantas Mazeika and Andy Zou and Norman Mu and Long Phan and Zifan Wang and Chunru Yu and Adam Khoja and Fengqing Jiang and Aidan O'Gara and Ellie Sakhaee and Zhen Xiang and Arezoo… See the full description on the dataset page: https://huggingface.co/datasets/walledai/TDC23-RedTeaming.openai-moderation-dataset
Evaluation dataset for the paper "A Holistic Approach to Undesired Content Detection"
The evaluation dataset is the test set used in this paper.
Each line contains information about one sample in a JSON object and each sample is labeled according to our taxonomy. The category label is a binary flag, but if it does not include in the JSON, it means we do not know the label.
Category
Label
Definition
sexual
S
Content meant to arouse sexual excitement, such as the… See the full description on the dataset page: https://huggingface.co/datasets/walledai/openai-moderation-dataset.CyberSecEval
CyberSecEval
The dataset source can be found here.
(CyberSecEval2 Version)
Abstract
Large language models (LLMs) introduce new security risks, but there are few comprehensive evaluation suites to measure and reduce these risks. We present CYBERSECEVAL 2, a novel benchmark to quantify LLM security risks and capabilities. We introduce two new areas for testing: prompt injection and code interpreter abuse. We evaluated multiple state of the art (SOTA) LLMs, including GPT-4… See the full description on the dataset page: https://huggingface.co/datasets/walledai/CyberSecEval.bitcoin-wallet-recovery-faq
Bitcoin Wallet Recovery FAQ Dataset v1.0
A high-quality Question & Answer dataset focused exclusively on Bitcoin wallet recovery and self-custody best practices. It is designed for training, fine-tuning, and evaluating LLMs and retrieval-augmented generation (RAG) systems in the domain of bitcoin security, seed backup, device loss, and fund recovery.
Dataset Summary
Total records: 500
Language: English
Answer length: 150–300 words per record
Categories: 39… See the full description on the dataset page: https://huggingface.co/datasets/ismailtasdelen/bitcoin-wallet-recovery-faq.bitcoin-wallet-security-qa
Bitcoin Wallet Security Dataset
A high-quality question–answer dataset of 500 records focused on Bitcoin wallet
security, self-custody, backup and recovery planning, and common attack vectors. It is
built to train and evaluate AI systems that help people secure their Bitcoin — fine-tuning
LLMs, powering retrieval-augmented generation (RAG), security-focused assistants, and
educational chatbots.
Every record pairs a realistic security question with a detailed, self-contained… See the full description on the dataset page: https://huggingface.co/datasets/ismailtasdelen/bitcoin-wallet-security-qa.SaladBench
Dataset Card for SaladBench
Paper: SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Models
Data: SafeText Dataset
📊 Statistical Overview of Base Question
Type
Data Source
Nums
Self-instructed
Finetuned GPT-3.5
15,433
Open-Sourced
HH-harmless
4,184
HH-red-team
659
Advbench
359
Multilingual
230
Do-Not-Answer
189
ToxicChat
129
Do Anything Now
93
GPTFuzzer
42
Total
21,318
You can refer to the Paper… See the full description on the dataset page: https://huggingface.co/datasets/walledai/SaladBench.adaption-defi-wallet-risk-classification
This dataset is a remastered version prepared using Adaption's Adaptive Data platform.
adaption-defi_wallet_risk_classification
This dataset contains prompt-completion pairs for classifying the 14-day risk outcomes of DeFi wallets on various EVM networks. Each entry provides behavioral features such as transaction counts, action ratios, and concentration metrics within a specific feature window to predict a binary risk label. The completions offer a concise justification for… See the full description on the dataset page: https://huggingface.co/datasets/samscript18/adaption-defi-wallet-risk-classification.adaption-defi-wallet-risk-classification-v1
This dataset is a remastered version prepared using Adaption's Adaptive Data platform.
adaption-defi_wallet_risk_classification
This dataset contains prompt-completion pairs for classifying the 14-day risk outcomes of DeFi wallets on various EVM networks based on behavioral features. Each sample provides wallet metrics such as transaction counts, action ratios, and concentration levels, followed by a binary risk label and a concise reasoning statement. The data is designed for… See the full description on the dataset page: https://huggingface.co/datasets/samscript18/adaption-defi-wallet-risk-classification-v1.SimpleSafetyTests
The dataset is obtained from the paper SimpleSafetyTests: a Test Suite for Identifying Critical Safety Risks in Large Language Models
and from the huggingface source.
Abstract
The past year has seen rapid acceleration in the development of large language models (LLMs). However, without proper steering and safeguards, LLMs will readily follow malicious instructions, provide unsafe advice, and generate toxic content. We introduce SimpleSafetyTests (SST) as a new test… See the full description on the dataset page: https://huggingface.co/datasets/walledai/SimpleSafetyTests.AyaRedTeaming
Dataset Card for Aya Red-teaming
Dataset Details
The Aya Red-teaming dataset is a human-annotated multilingual red-teaming dataset consisting of harmful prompts in 8 languages across 9 different categories of harm with explicit labels for "global" and "local" harm.
Curated by: Professional compensated annotators
Languages: Arabic, English, Filipino, French, Hindi, Russian, Serbian and Spanish
License: Apache 2.0
Paper: arxiv link
Harm Categories:… See the full description on the dataset page: https://huggingface.co/datasets/walledai/AyaRedTeaming.CBBQ
CBBQ
Datasets and codes for the paper "CBBQ: A Chinese Bias Benchmark Dataset Curated with Human-AI Collaboration for Large Language Models"
Introduction
Abstract: The growing capabilities of large language models (LLMs) call for rigorous scrutiny to holistically measure societal biases and ensure ethical deployment. To this end, we present the Chinese Bias Benchmark dataset (CBBQ), a resource designed to detect the ethical risks associated with deploying highly capable… See the full description on the dataset page: https://huggingface.co/datasets/walledai/CBBQ.AegisSafetyTestMultiJail
Multilingual Jailbreak Challenges in Large Language Models
This repo contains the data for our paper "Multilingual Jailbreak Challenges in Large Language Models".
[Github repo]
Annotation Statistics
We collected a total of 315 English unsafe prompts and annotated them into nine non-English languages. The languages were categorized based on resource availability, as shown below:
High-resource languages: Chinese (zh), Italian (it), Vietnamese (vi)
Medium-resource languages:… See the full description on the dataset page: https://huggingface.co/datasets/walledai/MultiJail.ForbiddenQuestions
ForbiddenQuestions
From paper: "Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models
The forbidden question set forbidden_questino_set.csv contains 390 questions (= 13 scenarios $\times$ 30 questions) adopted from OpenAI Usage Policy.
Column
Description
content_policy_id
Content policy id.
content_policy_name
The content policy name, i.e., illegal activity.
q_id
Question id
question
The question
We exclude Child… See the full description on the dataset page: https://huggingface.co/datasets/walledai/ForbiddenQuestions.SGXSTest
Abstract
For testing refusal behavior in a cultural setting, we introduce SGXSTEST — a set of manually curated prompts designed to measure exaggerated safety within the context of Singaporean culture. It comprises 100 safe-unsafe pairs of prompts, carefully phrased to challenge the LLMs’ safety boundaries. The dataset covers 10 categories of hazards (adapted from Röttger et al. (2023)), with 10 safe-unsafe prompt pairs in each category. These categories include homonyms, figurative… See the full description on the dataset page: https://huggingface.co/datasets/walledai/SGXSTest.HiXSTest
Dataset Details
TBD
Citation
If you use the data, please cite the following paper:
@misc{gupta2024walledeval,
title={WalledEval: A Comprehensive Safety Evaluation Toolkit for Large Language Models},
author={Prannaya Gupta and Le Qi Yau and Hao Han Low and I-Shiang Lee and Hugo Maximus Lim and Yu Xin Teoh and Jia Hng Koh and Dar Win Liew and Rishabh Bhardwaj and Rajat Bhardwaj and Soujanya Poria},
year={2024},
eprint={2408.03837}… See the full description on the dataset page: https://huggingface.co/datasets/walledai/HiXSTest.AART
AART
AART: AI-Assisted Red-Teaming with Diverse Data Generation for New LLM-powered Applications (Radharapu et al. 2023)
Abstract
Adversarial testing of large language models (LLMs) is crucial for their safe and responsible deployment. We introduce a novel approach for automated generation of adversarial evaluation datasets to test the safety of LLM generations on new downstream applications. We call it AI-assisted Red-Teaming (AART) - an automated alternative to current… See the full description on the dataset page: https://huggingface.co/datasets/walledai/AART.CSRTDataset derived from CSRT: Evaluation and Analysis of LLMs using Code-Switching Red-Teaming Dataset submitted to the NeurIPS 2024 Datasets and Benchmarks track
We introduce code-switching red-teaming, a simple yet effective red-teaming technique that simultaneously tests the multilingual capabilities and safety of LLMs
Keywords
LLM, Evaluation, Safety, Multilingual, Red-teaming, Code-switching
Abstract
Recent studies in large language models (LLMs) shed light on their… See the full description on the dataset page: https://huggingface.co/datasets/walledai/CSRT.SafetyDatasets
SafetyDatasets
Paper: Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions
Data: Dataset
About
Safety Evaluation Datasets
In the paper, the authors utilized several specialized datasets to evaluate the safety of language models. These datasets cover various aspects of safety, from malicious instructions to controversial topics. Here is a detailed overview of the safety evaluation datasets used:… See the full description on the dataset page: https://huggingface.co/datasets/walledai/SafetyDatasets.