xilinghuiye/RoomReader-Full

RoomReader-Full

RoomReader-Full is a multimodal benchmark for fine-grained understanding of how a speaker manages information during a conversation. Given a short target video clip and its audio/dialogue, a model must determine whether the response is consistent with the relevant interactional expectation and assign one of six information strategy labels.

The benchmark is designed for cases where the observable response alone is not enough to determine the label. Formal gold construction uses source-grounded truth, speaker knowledge, interactional expectation, and temporally aligned audiovisual evidence. The rich source context is used during annotation; the evaluation input is the released target clip and the fields selected by an experimental protocol.

Dataset Summary

The two domains provide complementary difficulty regimes:

• —Scripted film / easy: target interactions are grounded in a complete film and an independent screenplay, literary source, or equivalent source-of-truth text.
• —Werewolf / hard: target interactions come from social-deduction games with asymmetric role information, game transcripts, and complete single-game videos.

Label Taxonomy

The six labels form a two-level task. The coarse label indicates whether the observed behavior is consistent with the relevant expectation; the fine label describes the information strategy.

File Layout

The Hugging Face repository currently contains the portable annotation tables and label/split metadata:

README.md
data/
  annotations/
    all/
      part-*.parquet
    train/
      part-*.parquet
    test/
      part-*.parquet
metadata/
  label_mapping.json
  split_policy.json

The Parquet tables contain the complete annotation metadata and portable relative media references. Each split is stored as a directory of small Parquet shards so that the dataset can be downloaded through standard Hugging Face file transfer without requiring a single large multipart object. A separately validated self-contained media archive contains the released video and visual media bytes. The archive is 43.5 GB and contains 18,237 film clips, their silent variants and visual inspection assets, plus 151 complete werewolf game videos. Its release path and exact checksum are maintained with the benchmark release artifacts because the current mirror's large-file upload endpoint does not complete reliably for this archive.

Loading the Parquet Shards

Each directory is a complete logical table. Use the all directory for the full dataset, or use the train and test directories independently:

from pathlib import Path

import pandas as pd

root = Path("/path/to/cloned/RoomReader-Full")
parts = sorted((root / "data/annotations/test").glob("part-*.parquet"))
df = pd.concat((pd.read_parquet(path) for path in parts), ignore_index=True)
print(len(df))  # 4,022

The shard files have the same schema and together contain 29,480 unique samples across all, 25,458 samples across train, and 4,022 samples across test.

Annotation Fields

The most important fields in the Parquet shards are:

• —sample_id: globally unique sample identifier;
• —split: train or test;
• —source_domain: scripted_film or werewolf_social_deduction;
• —difficulty_band: easy or hard;
• —fine_label, fine_label_zh: six-way annotation;
• —consistency: coarse consistent / inconsistent label;
• —truth_relation: matches, partial, leaks, contradicts, evades, or counterclaims;
• —target_person, target_role: target speaker identity and role information when available;
• —target_response, trigger_context, next_context: dialogue context;
• —truth, rationale: source-grounded truth and annotation rationale;
• —clip_start_sec, clip_end_sec, clip_duration_sec, target_timestamp_sec: temporal localization;
• —video_path, silent_video_path, thumbnail_path, contact_sheet_path: paths relative to the release root;
• —media_locator_json: portable media and temporal locator;
• —source_record_json: source-specific annotation audit and provenance record;
• —quality_tier, confidence_score, gold_status: quality and formal-gold status.

Relative Media Paths

All media paths in the released annotations are relative to the root of the validated media package. No /mnt/..., /home/..., hf_parquet://, or other machine-local path is required. The Parquet tables therefore remain portable even when the media archive is downloaded separately.

For a scripted-film row, video_path points directly to the short target clip:

media/film_clips/train/<sample_id>.mp4

The corresponding audio-free clip is available at silent_video_path. The visual inspection assets are available at thumbnail_path and contact_sheet_path.

For a werewolf row, video_path points to a complete single-game video:

media/werewolf_videos/Youtube/videos/<source-video-name>.mp4

Use clip_start_sec and clip_end_sec to extract the annotated target moment from that complete video. The release includes 151 complete game videos referenced by the 11,243 hard-domain rows.

Example loader:

import json
from pathlib import Path

import pandas as pd

root = Path("/path/to/cloned/RoomReader-Full")
parts = sorted((root / "data/annotations/test").glob("part-*.parquet"))
df = pd.concat((pd.read_parquet(path) for path in parts), ignore_index=True)
row = df.iloc[0]
video = root / row.video_path
print(row.fine_label, row.consistency, video)

For werewolf rows, the video path is the complete game video. A media reader should use the row's clip_start_sec and clip_end_sec for temporal slicing.

Split Protocol

There is no development split in this release. The test split is designed to reduce source leakage:

• —film test examples are held out by complete film source;
• —werewolf test examples are held out by complete single-game video;
• —RoomReader-Core-Test is exactly the test portion of RoomReader-Full.

The exact source-held-out policy and validation report are in metadata/split_policy.json.

Formal Gold Construction

Formal gold labels are not bootstrap model predictions and are not generated from subtitle keywords alone. For scripted films, the source-specific annotation module uses a complete original film, an independent source-of-truth text, explicit truth groups, and explicit subtitle-index specifications. Subtitles provide temporal alignment, while the independent source establishes narrative truth. For werewolf data, role truth, game state, transcript context, and the complete source video are used to ground the annotation.

The annotation audit records the source-specific truth graph, speaker knowledge, environment expectation, response relation, media cut, and validation status in source_record_json. The formal gold status is recorded in gold_status.

Provenance and Rights

The release contains source-specific provenance rather than a single blanket license. Consult metadata/source_manifest.jsonl before redistribution or commercial use.

The scripted-film sources in the current package were acquired from public-domain or public-domain-equivalent sources and include source-specific provenance URLs. The werewolf videos originate from public online uploads and are included as benchmark media references in this research release; users must independently verify the rights and terms for each upstream video before redistribution. The dataset annotations, source-specific annotation code, and benchmark schema are provided for research and evaluation purposes.

Release Status

The following files are published in this repository and can be loaded directly:

• —data/annotations/all/part-*.parquet;
• —data/annotations/train/part-*.parquet;
• —data/annotations/test/part-*.parquet;
• —metadata/label_mapping.json;
• —metadata/split_policy.json.

The complete local media archive was validated with ZIP CRC testing and has no absolute paths in its annotation files. Uploading that 43.5 GB archive through the current mainland mirror reached the large-file transfer stage but failed during the mirror's multipart completion request. The annotation Parquet upload itself completed successfully.

Citation

Please cite the accompanying InfoCon / RoomReader benchmark paper when using this dataset. The final citation will be added here when the paper metadata is finalized.

Contact

Please use the Hugging Face dataset discussion page for issues concerning loading, annotation fields, split policy, or provenance.