redmadrobot-rnd/nsfw_benchmark

Guardrail Model Evaluation Dataset

Dataset Description

This dataset is designed for evaluating AI safety systems (guardrails), NSFW (Not Safe For Work) content classification, and toxicity detection systems.

The test set includes challenging real-world examples, collected and labeled manually. The dataset is available in two versions: the original Russian version and an English translation. It allows you to evaluate models on complex and borderline examples. We position this dataset as a robust benchmark, and a comparative evaluation of several prominent safeguard systems is presented below.

Content Categories

The dataset is annotated across 7 main categories of harmful content:

1. Violence

• —Physical violence and aggression: Injury, use of weapons, fighting, intimidation, and vandalism intended to intimidate.
• —Murder and cruelty: Deprivation of life, mention of corpses, sadism, torture, dismemberment, and dehumanization.
• —Mass violence: War crimes, terrorist attacks, genocide, ethnic cleansing, and mass shootings.
• —Justification of violence: Normalization of cruelty, victim blaming, and glorification of criminals.

2. Erotic and Pornography

• —Sexual scenarios: Descriptions of sexual intercourse with the explicit purpose of arousal.
• —Artistic erotica: Detailed descriptions of foreplay, caresses, kisses, and erotic tension.
• —Pornography (Explicit): Graphic descriptions of sexual intercourse, genitals, bodily secretions, and the mechanics of orgasm.

3. Animal Cruelty

• —Sadism and violence: Intentional infliction of pain, injury, torture, and abuse of animals.
• —Abandonment in danger: Deprivation of food/water, denial of treatment, keeping in critical conditions, and abandonment.
• —Bloodfights and baiting: Organizing dog/cockfights, baiting animals with live bait for the sake of spectacle.

4. Drug Abuse

• —Use and Consequences: Description of the drug use process, states of euphoria/hallucination, withdrawal symptoms, and overdoses.
• —Production and Distribution: Instructions for manufacturing, diagrams of drug drop locations, couriers, and darknet traffic.
• —Storage and Paraphernalia: Display of substances, paraphernalia, and evidence related to distribution.

5. Self-Harm/Suicide

• —Self-Harm: Direct physical injury to oneself (cuts, burns, blows).
• —Eating Disorders and Exhaustion: Promotion of anorexia, bulimia, dangerous diets, and sleep deprivation.
• —Health Sabotage: Intentional non-fatal overdoses, poisoning, or refusing life-saving treatment.
• —Destructive Risk: Victimizing behavior or intentionally seeking extreme trauma.

6. Politics / Hate Speech

• —Disinformation: Spreading election fakes, deepfakes of politicians, and slander against institutions.
• —Campaigning: Political PR, direct calls to vote, and promoting partisanship.
• —Hate Speech: Dehumanizing opponents, inciting civil hatred, and calling for politically motivated repression.

7. Extremism

• —Political/Social Extremism: Calls for uprisings, coups, and the elimination of opponents.
• —Racism and Nazism: Propaganda of neo-Nazism, ethnic cleansing, and the use of prohibited symbols.
• —Radicalism and Terrorism: Justification of terrorist attacks, activities of destructive sects, calls for violent jihad and eco-sabotage.
• —Cyberterrorism: Instructions for hacking critical infrastructure, creating weapons from blueprints, and online recruitment.

Test Sample Composition:

Table 1. Distribution of examples by category and safety label

The dataset is fully bilingual. All original examples were collected in Russian and subsequently translated into English.

Table 2. Distribution of examples by category and language

The dataset consists of three main types of data designed to rigorously evaluate model capabilities:

1. 1.Original unsafe texts (Unsafe / Original): Real user queries, questions, and requests that were collected, verified, and manually labeled across all 7 categories.
2. 2.Contrastive safe texts (Contrastive / Safe): Created directly from the unsafe examples from the first point. Using the gemini-2.5-flash model, key details were changed in the original texts, turning the query into a safe one. These paired examples were added specifically to evaluate the models for vulnerability to shortcut learning (memorizing false features). All generated texts have undergone rigorous manual verification and correction.
3. 3.Hard Negatives / Safe Texts: "Borderline" examples that were found and filtered entirely manually.

Table 3. Breakdown of safe examples by type

Data Structure

(Note: Each text is assigned only one, most relevant category)

Benchmarks

This dataset serves as a robust benchmark for guardrail model evaluation. Below are the baseline evaluation results for several prominent open-weight models.

Evaluation Metric: F1-score (macro)

Evaluated Models & Hardware Setup:

• —[Llama Guard 3 8B (Q8_0)](https://huggingface.co/meta-llama/Llama-Guard-3-8B)
• —[Qwen3Guard Gen 8B (Q8_0)](https://huggingface.co/Qwen/Qwen3Guard-Gen-8B)
• —[gpt-oss-120b](https://huggingface.co/openai/gpt-oss-120b): via a custom safeguard prompt

Taxonomy Mapping

To ensure a fair comparison, we mapped our dataset's 7 categories to the specific safety taxonomies defined by the tested models:

(Note: For detailed descriptions of these classes, please refer to the official documentation of the respective models).

Results: F1-score Macro (%)

English Subset (EN)

Russian Subset (RU)