BushNate/llm-refusal-evaluation

🛡️ LLM Refusal Evaluation Benchmark

This repository contains the benchmarks used in the [LLM-Refusal-Evaluation](https://github.com/CompactifAI/LLM-Refusal-Evaluation) suite.

The prompts are organized into three groups:

• —Safety Benchmarks — harmful / jailbreak-style prompts that models should refuse.
• —Chinese Sensitive Topics — prompts that may be censored by China-aligned models.
• —Sanity Check Datasets — non-sensitive prompts to ensure models don’t over-refuse.

📌 Contents

• —Safety Benchmarks
• —JailbreakBench
• —SorryBench
• —XSTest
• —HarmBench (Sampled)
• —Adversarial Unsafe Prompts
• —Chinese Sensitive Topics
• —CCP Sensitive
• —DeCCP
• —Sanity Check Datasets
• —General Prompts

🧪 Safety Benchmarks

JailbreakBench

Source: https://arxiv.org/abs/2404.01318

JailbreakBench is an open-source robustness benchmark for jailbreaking large language models (LLMs). It aims to comprehensively track progress toward:

1. 1.Generating successful jailbreaks
2. 2.Defending against jailbreak attempts

SorryBench

Source: https://arxiv.org/abs/2406.14598

A dataset of 440 unsafe instructions, spanning 44 fine-grained safety categories (10 examples per category). It is designed to be balanced across categories, unlike many prior safety datasets.

XSTest

Source: https://arxiv.org/abs/2308.01263

XSTest includes two splits:

• —`xtest_safe`: 250 safe prompts across 10 prompt types that well-calibrated models should not refuse.
• —`xtest_unsafe`: 200 unsafe prompts that most LLM applications should refuse.

HarmBench (Sampled)

Source: https://arxiv.org/abs/2402.04249

A collection of harmful prompts created for red-teaming evaluation. We sample 256 prompts across all categories.

Adversarial Unsafe Prompts

Source: https://huggingface.co/datasets/nvidia/Nemotron-Content-Safety-Reasoning-Dataset Method: Automatic injection of jailbreak prompts

Following [How to jailbreak DeepSeek R1 1776](https://weijiexu.com/posts/jailbreak_How%20to%20jailbreak%20DeepSeek%20R1%201776r1_1776.html), we design a dataset that combines harmful prompts with Chinese sensitive-topic injections.

This evaluates whether “uncensored” models:

• —trivially answer any request containing China, or
• —actually maintain safety alignment and refuse harmful requests even when sensitive-topic content is injected.

✅ Expected behavior: models should refuse these prompts.

🇨🇳 Chinese Sensitive Topics

CCP Sensitive

Source: https://huggingface.co/datasets/promptfoo/CCP-sensitive-prompts

Prompts covering sensitive topics in China that are likely to be censored by China-aligned models.

DeCCP

Source: https://huggingface.co/datasets/augmxnt/deccp

Prompts about Chinese sensitive topics that trigger refusal responses in Qwen2-7B-Instruct.

✅ Sanity Check Datasets

General Prompts

Source: https://huggingface.co/datasets/HuggingFaceTB/smoltalk

A set of 100 random non-sensitive prompts (math, coding, summarization, etc.) used to verify that models aren’t over-refusing benign requests.