Yuan4629/WereBench

Anonymization

For all content in this Hugging Face dataset repository and GitHub repository, we have ensured that anonymization has been performed, making it impossible to trace back to the authors' information.

WereBench

WereBench is a benchmark dataset for evaluating language models in the Werewolf (similar to Mafia) social deduction setting. It focuses on human‑aligned strategic reasoning rather than only coarse metrics (e.g., win rate), aligning model behavior with successful human strategies. This repository hosts the data that will be uploaded to the Hugging Face Hub at datasets/Author/WereBench. The primary entry file is /data/WereBench.jsonl, and additional raw materials live under data/Season_*. We also provide WereBench.json to help understand WereBench.

What’s Inside

• —WereBench.json: the main question set used for speech‑level evaluation and analysis.
• —json2jsonl.py: the script we used to convert WereBench.json to WereBench.jsonl.
• —data/WereBench.jsonl: WereBench.json in jsonl format for machine readability.
• —data/Season_*/*.json: supporting materials referenced by questions (dialogue/episode fragments and metadata).
• —data/video: video dataset of werewolf TV programs, Panda Kill.

Schema of WereBench.json

The file is a structured JSON object with the following top‑level fields:

• —questionSetTitle: title of the full question set.
• —version: dataset version string.
• —categories: list of category objects:
• —categoryName: name of the category (e.g., Role Inference, Strategic Judgment, Deception Reasoning, Persuasive Statements, Counterfactual Trade‑off).
• —questions: list of question objects with fields:
• —id: question id (int)
• —utterance_id: link to a referenced utterance/segment (int)
• —role: target role perspective for the question (string)
• —text/text1: prompt in natural language
• —options: list of multiple‑choice options, each with key (e.g., "A") and text
• —answerKey: the correct option key (e.g., "H")
• —reference: relative path to supportive material (e.g., data/Season_X/YY.json)
• —explanation: optional rationale/explanation (string, may be empty)

Note: Most of the content is in Chinese because our dataset comes from real human gameplay scenarios common in Chinese Werewolf communities.

Quick Start

You can load WereBench.json directly via the datasets library as a JSON file.

from datasets import load_dataset

ds = load_dataset(
    "json",
    data_files="https://huggingface.co/datasets/Author/WereBench/resolve/main/data/WereBench.jsonl"
)

To work with all supplementary JSON files under data/, you can also load a pattern or multiple files:

from datasets import load_dataset

ds = load_dataset(
    "json",
    data_files={
        "support": [
            "https://huggingface.co/datasets/Author/WereBench/resolve/main/data/Season_1/01.json",
            # ... add the files you need or script a crawler
        ]
    }
)

Evaluation Dimensions (WereAlign)

Following the WereBench/WereAlign framework:

• —Role Inference (RI): infer identities/intentions of other players.
• —Strategic Judgment (SJ): choose actions beneficial to own faction.
• —Deception Reasoning (DR): detect deception or perform masquerade.
• —Persuasive Statements (PS): produce convincing statements in context.
• —Counterfactual Trade‑off (CT): weigh risks/benefits across alternatives.

The dataset supports speech‑level multiple‑choice evaluation and can be extended to decision‑level analyses (e.g., vote alignment, opponent‑role inference) when paired with game logs.

Repository and Paper

• —Code and broader benchmark context: GITHUB_REPO
• —Paper (arXiv): ARXIV_LINK

Citation

If you use WereBench, please cite the project and the associated paper: