web3w/role-play-bench

Role-play Benchmark

A comprehensive benchmark for evaluating Role-play Agents in Chinese and English scenarios.

Dataset Summary

Role-play Benchmark is designed to evaluate Role-play Agents' ability to deliver immersive role-play experiences through Situated Reenactment. Unlike traditional benchmarks with verifiable answers, Role-play is fundamentally non-verifiable, e.g., there's no single "correct" response when a tsundere character is asked "Do you like me?". Instead, we focus on detecting misalignment: identifying what clearly does NOT fit the character setting.

The benchmark addresses three core challenges in Role-play Agents:

• —Worlds (Consistency & Logic): Evaluating the agent's ability to eliminate technical glitches and "catastrophic forgetting," ensuring long-term adherence to established relationships and world laws to prevent immersion-breaking contradictions.

• —Stories (Narrative Progression): Assessing whether the agent drives the plot forward with semantic variety and motivated character evolution, avoiding repetitive patterns, stagnation, or out-of-character (OOC) behaviors.

• —User Preferences (Interaction Quality): Testing the agent’s interactive boundaries and responsiveness—ensuring it avoids speaking for the user, responds to context, and consistently provides "hooks" to invite further engagement.

Key Features

• —Situated Reenactment: Evaluates agent performance at specific {World} × {Stories} coordinates for given {User Preferences}
• —Model-on-Model Self-Play: Uses a carefully crafted User Simulator to generate multi-turn dialogue trajectories
• —Negative Evaluation (Flooring): Focuses on detecting misalignment rather than defining "perfect" responses
• —Multi-Dimensional Scoring: Combines Basics, Logic, Knowledge, Diversity, Content Logic, and Interaction metrics

Data Source

Data Source, Privacy & Scope

All dialogues in this release are synthetically constructed by our data team. This is a partial subset of the full benchmark, and the released samples are intentionally skewed toward a chat-oriented role-play style, designed to mirror online user interaction patterns in conversational settings.

Controlled Variations & Robustness Evaluation

To better assess robustness, the dataset intentionally includes controlled perturbations in input formats, writing styles, and prompt structures. We also introduce light variations to selected user-side fields (e.g., initialuserinput, user_setting) in parts of the data. A key evaluation objective is output stability—whether models can maintain consistent, high-quality role-play behavior under noisy or non-standard inputs.

Design

We explored multiple situated reenactment designs to reflect different ways role-play scenarios unfold in practice, and use two complementary datasets (en/zh) to showcase these design choices under controlled vs. diverse user-side conditions.

• —English (en): We standardize user-side fields (e.g., username, usersetting, initialuserinput) to isolate NPC performance and make cross-NPC comparisons cleaner.
• —Chinese (zh): We allow scenario-dependent variations in usersetting and initialuser_input to reflect diverse user personas and different conversation starting points, enabling evaluation of model behavior across a broader range of initial conditions.

Dataset Structure

Configurations

This dataset has 2 configurations:

Data Fields

Seeds Table (seeds_zh / seeds_en)

Role-play scenario seeds containing character settings and initial context.

Dialogues Table (dialogues_zh / dialogues_en)

Generated multi-turn dialogue trajectories from agent self-play.

Evaluations Table (evaluations_zh / evaluations_en)

Session-level evaluation scores across 6 dimensions (0-100 scale).

Evaluation Metrics

The evaluation directly maps to the three core objectives:

Detailed Dimensions

Worlds Dimensions:

• —Basics: Text quality — avoiding garbled text, language mixing, severe repetition
• —Logic: Logical consistency — preventing character confusion, spatial errors, pronoun reference confusion
• —Knowledge: Factual reliability — balancing rules between virtual and real worlds

Stories Dimensions:

• —Diversity: Expression variety and narrative momentum — penalizing dialogue stagnation and mechanical loops
• —Content Logic: Narrative coherence and OOC (Out-of-Character) detection — allowing character evolution with proper buildup

User Preferences Dimensions:

• —Interaction: Penalizes behaviors like AI Speaks for User (acting on user's behalf), AI Ignores User (self-talking), and Over Refusal (excessive rejection of interaction).

Evaluation Protocol

• —Long-Context Interaction: Each session consists of 100 dialogue turns to test stability and consistency over time.
• —Multi-Sampling: We conduct 3 independent sessions per scenario to account for generation variance.
• —Chunked Judging: Conversations are sliced into 20-turn chunks for stable LLM-as-Judge evaluation.
• —Human Calibration: The automated metrics are regularly calibrated against human preference data to ensure alignment.

Leaderboard

Overall Rankings

📖 Citation

@dataset{MiniMaxAI_role-play_bench_2026,
  title={Role-play Benchmark},
  author={MiniMax},
  year={2026},
  publisher={Hugging Face},
  url={https://huggingface.co/datasets/MiniMaxAI/role-play-bench}
}

📄 License

CC BY 4.0 / CC BY-SA 4.0