SamsungSDS-Research/Policy-on-the-Fly-Benchmark

⚠️ Content Warning: This dataset contains harmful content for AI filter evaluation, including biased expressions, crime-related scenarios, and jailbreak attempts.

PoFBench: Policy-on-the-Fly Benchmark

Overview

PoFBench (Policy-on-the-Fly Benchmark) is a test-only benchmark designed to measure the performance of policy-based custom filters in LLM-powered systems.

Existing AI safety benchmarks evaluate against fixed risk taxonomies predefined by experts. However, in real-world enterprise environments, the same text may need to be blocked or allowed depending on the specific policy applied.

PoFBench measures whether a model consistently changes its verdict when only the policy combination is changed for the same prompt.

• —📊 1,080 instances — 540 Korean (original) + 540 English (translated)
• —🏷️ 8 policy categories × 6 policy combinations
• —🎯 Block 33% / Allow 67% label distribution
• —📋 Test-only — No train/validation splits provided

📄 For research background and detailed methodology, please refer to our paper (available soon).

Target Systems

PoFBench can be used to evaluate the performance of policy-based filter (guardrail) systems operating in the following environments:

• —LLM input/output filtering systems
• —AI agent guardrails
• —Chatbot content filters
• —RAG system safety filters

Dataset Structure

Fields

Each row represents a single evaluation sample, consisting of a prompt, a policy, and the expected verdict for that combination.

The policy column is a structured text composed of four sections:

• —Policy Name — Name of the policy
• —Description — Description of the topic this policy monitors
• —Block Condition — Specific blocking criteria. Only content matching these conditions should be blocked
• —Allow Condition — Exception criteria (exemption). Only content matching these conditions should be allowed. Set to "None" if not applicable

Policy Categories

The dataset covers 8 risk categories that can arise during LLM usage. Each category reflects a security policy domain commonly operated in enterprise environments.

Combination Cases

This is the core design of PoFBench. For each prompt, 6 evaluation samples are generated by varying the positive/negative combinations of block and allow conditions. This allows distinguishing whether a model actually interprets the policy content or merely judges based on the prompt alone.

• —positive_block — A policy whose block condition matches the prompt
• —negative_block — A policy whose block condition does not match the prompt
• —positive_allow — A policy whose allow condition matches the prompt → block exemption
• —negative_allow — A policy whose allow condition does not match the prompt

Multi-Policy Evaluation

PoFBench supports multi-policy configuration, offering up to 100 policies per prompt. This design evaluates a model's ability to:

• —Assess relevance between a prompt and both related and unrelated policies
• —Handle long-context reasoning when presented with up to 100 policies simultaneously

Each data entry contains the original prompt with its corresponding block and allow conditions, along with an unaffected_policy_list column that holds 99 additional policies which do not affect the original prompt-policy label.

To preserve label consistency, users may select the original prompt + policy and freely choose how many policies to include from the unaffected_policy_list column.

⚠️ Among the 540 entries per language, 5 entries were found to be heavily influenced by other policies and therefore do not fit the multi-policy scenario. Their unaffected_policy_list column contains an empty list ([]).

Content Safety

PoFBench intentionally includes harmful content for the purpose of AI filter evaluation. The table below shows the presence and severity of harmful content by category.

The following types of content are NOT included in this dataset: child sexual abuse material (CSAM), self-harm/suicide, weapons/drug manufacturing instructions, content targeting real individuals, personally identifiable information (PII)

Usage

from datasets import load_dataset

# English
dataset_en = load_dataset("SamsungSDS-Research/Policy-on-the-Fly-Benchmark", "pofbench_en", split="test")

# Korean
dataset_ko = load_dataset("SamsungSDS-Research/Policy-on-the-Fly-Benchmark", "pofbench_ko", split="test")


# Multi-Policy Evaluation
# Convert the 'policy' column of df into a list that can be indexed
df = dataset_ko.to_pandas()   
all_policies = df['policy'].tolist()

# Map the indices in 'unaffected_policy_list' to the actual policy strings
df['unaffected_policy_list_text'] = df['unaffected_policy_list'].apply(
    lambda idx_list: [all_policies[i] for i in idx_list]
)

Data Creation

PoFBench was created by experts from the SDS AI division between October and December 2025. No external user data was collected or crawled, and all scenarios are fictional with no personally identifiable information (PII) included. Prompts were directly authored in various formats including general questions, agent calls, and RAG usage patterns. Policies were generated using Qwen3-235B(Apache 2.0) and subsequently verified through human review. To ensure data quality, three models (GPT-4.0, Gemma3-27B, Qwen3-235B) were used for cross-validation, with policies adjusted until at least 2 out of 3 models agreed on the label. Human reviewers then refined the prompts and policies in detail. English data was translated from Korean using gemma3-27B, followed by human review. This dataset does not involve external human subjects research and is exempt from IRB review. All creators participated voluntarily, and session time limits were applied when authoring harmful content. For detailed information on the data creation pipeline and validation process, please refer to our paper.

Intended Use & Limitations

Intended Use

PoFBench is designed for evaluating the performance of AI custom filters and guardrail systems. It can be used for policy-based content filtering research, pre/post filter verification in LLM, agent, chatbot, and RAG systems, and analyzing filter performance variance across policy changes.

Limitations

• —As English data are translated from Korean, slight differences in linguistic and cultural nuance may exist
• —The 8 categories do not fully cover all possible enterprise security policies

Restrictions

By accessing this dataset, you agree:

• —No Commercial Use – You may not use the dataset for any commercial purpose.
• —No Harmful Use – You may not use the dataset to intentionally generate, promote, or distribute harmful, offensive, or unsafe content.
• —Research Only – Usage is restricted to non-commercial research and educational purposes.
• —No Redistribution – You may not redistribute the dataset in whole or in part without permission.
• —Compliance – You must comply with all applicable laws and regulations regarding data usage.

Disclaimer

• —The dataset contains harmful and offensive language.
• —The dataset is provided "as is", without warranties or guarantees.
• —The authors and hosting platforms (e.g., Hugging Face) are not responsible for any misuse or damages resulting from the use of this dataset.

License

PoFBench is released under the CC-BY-NC-4.0 license. Due to the inclusion of harmful content, it is provided via Gated Access and can be downloaded after agreeing to the research-only use terms.

Citation

@misc{pofbench2026,
      title={Policy-on-the-Fly: A Principled Benchmark for Policy-Conditioned Compliance in AI Systems}, 
      author={JoonHo Lee and Hyeonmin Cho and HyunJae Lee and Juree Seok and Jaewoong Yun and JunKyu Lee and Seungki Hong},
      year={2026},
      eprint={TBD},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={TBD}, 
}