CoolFace
Datasetpublic

rrchen2026/Theory_of_Mind_CoMMET

CoMMET CoMMET is a multi-turn, multimodal benchmark designed to evaluate the Theory of Mind (ToM) capabilities of multimodal large language models (MLLMs). Unlike conventional single-turn Theory of Mind benchmarks, CoMMET represents each scenario as a sequence of interconnected turns. Models are required to reason about stories, previous interactions, feedback, questions, and, when necessary, visual information. The benchmark covers multiple types of mental-state reasoning and… See the full description on the dataset page: https://huggingface.co/datasets/rrchen2026/Theory_of_Mind_CoMMET.

sourceHugging Facemitupdated 15d agoView on Hugging Face
2likes156downloads
Dataset Card

CoMMET

CoMMET is a multi-turn, multimodal benchmark designed to evaluate the Theory of Mind (ToM) capabilities of multimodal large language models (MLLMs).

Unlike conventional single-turn Theory of Mind benchmarks, CoMMET represents each scenario as a sequence of interconnected turns. Models are required to reason about stories, previous interactions, feedback, questions, and, when necessary, visual information.

The benchmark covers multiple types of mental-state reasoning and is designed to evaluate whether models can maintain and update their understanding of characters' mental states across multiple conversational turns.

Dataset Overview

CoMMET contains:

  • —591 StoryTurns
  • —1,976 questions
  • —844 images

A StoryTurn represents a multi-turn reasoning scenario and may contain several questions. Later questions can depend on information introduced in earlier turns, including previous answers and feedback.

Dataset Structure

The dataset structure is as follows:

text
.
├── storyturn.parquet
└── images.parquet

1. StoryTurn

The file storyturn.parquet contains the structured benchmark data. Each row corresponds to one StoryTurn and contains the following fields:

FieldTypeDescription
storyturn_idint64Unique identifier of the StoryTurn.
contentlistA sequence of turns belonging to the StoryTurn. Each turn contains the story, question, answer, feedback, and associated image filename(s).
taskstringThe task category evaluated by the StoryTurn.
mental_stateslistMental state(s) evaluated in the StoryTurn, such as desire, belief, emotion, intention, etc.
image_requiredstringIndicates whether visual information is required to answer the questions correctly.
content

The content field contains a list of dictionaries. Each dictionary corresponds to one turn and follows the structure:

python
{
    "story": "...",
    "feedback": "...",
    "image_filename": [...],
    "question": "...",
    "answer": "..."
}

The fields are:

FieldDescription
storyStory or additional information introduced in the current turn. An empty string indicates that no new story information is introduced in this turn.
questionQuestion asked in the current turn.
answerReference answer for the question. For the Diverse Belief, Common Desire and Diverse Desire tasks, multiple sets of answers are acceptable. In such cases, the acceptable answers for each question are separated by a semicolon (;). Answers with the same index across different turns belong to the same valid answer set.
feedbackFeedback provided between turns, when applicable. An empty string indicates that no feedback is provided.
image_filenameA list containing the filename(s) of images associated with the current turn. An empty list indicates that no image is associated with the turn. When multiple valid answer sets are provided, a question may also be associated with multiple images. The appropriate image should be selected according to the answer index chosen for the first question in the multi-turn sequence.

For example:

python
{
    "storyturn_id": 0,
    "content": [
        {
            "story": "Here is Mia. Mia is deciding what to eat for a snack...",
            "question": "What do you like better - apples or bananas?",
            "answer": "apples",
            "feedback": "",
            "image_filename": ["storyturn_0_img_0.png"]
        },
        {
            "story": "",
            "question": "So which snack will Mia choose - the apple or the banana?",
            "answer": "apple",
            "feedback": "You do? That sounds delicious! Mia also likes that snack the best!",
            "image_filename": []
        }
    ],
    "task": "Common Desires",
    "mental_states": ["Desire"],
    "image_required": "No"
}

2. Images

The file images.parquet contains all images associated with the StoryTurn examples. Each row corresponds to one image and contains the following fields:

FieldTypeDescription
file_namestringFilename used to associate the image with the corresponding turn in storyturn.parquet.
imageImageThe image itself, stored directly in the Parquet file.

Images are linked to individual turns through the image_filename field in storyturn.parquet.

For example, if a turn contains:

python
"image_filename": ["storyturn_0_img_0.png"]

the corresponding row in images.parquet has:

python
{
    "file_name": "storyturn_0_img_0.png",
    "image": <image>
}

An empty list:

python
"image_filename": []

means that the corresponding turn does not contain an image.

Image filenames follow the convention:

text
storyturn_<storyturn_id>_img_<image_index>.png

For example:

text
storyturn_0_img_0.png
storyturn_100_img_0.png
storyturn_101_img_0.png
storyturn_101_img_1.png
storyturn_102_img_0.png

Usage

1. Loading the Dataset

To load the storyturn configuration:

python
from datasets import load_dataset

storyturn_dataset = load_dataset("rrchen2026/Theory_of_Mind_CoMMET", "storyturn")
sample = storyturn_dataset["data"][0]

print(sample["storyturn_id"])
print(sample["task"])
print(sample["mental_states"])

for turn in sample["content"]:
    print("Story:", turn["story"])
    print("Question:", turn["question"])
    print("Answer:", turn["answer"])
    print("Feedback:", turn["feedback"])
    print("Images:", turn["image_filename"])

To load the images configuration:

python
image_dataset = load_dataset("rrchen2026/Theory_of_Mind_CoMMET", "images")

image_sample = image_dataset["data"][0]

print(image_sample["file_name"])
print(image_sample["image"])

The image field is decoded by the Hugging Face datasets library as an image object.

2. Connecting StoryTurns with Images

The image_filename field in each StoryTurn can be used to retrieve the corresponding image from the images configuration.

For example:

python
storyturns = storyturn_dataset["data"]
images = image_dataset["data"]

# Map each image filename to its row index.
image_index = {
    file_name: i
    for i, file_name in enumerate(images["file_name"])
}

sample = storyturns[0]

for turn in sample["content"]:
    for image_filename in turn["image_filename"]:
        image = images[image_index[image_filename]]["image"]
        image.show()

3. Multi-Turn Evaluation

CoMMET is intended to be evaluated in a multi-turn setting.

For each StoryTurn, the elements in content should be processed sequentially:

text
Turn 1
  Story
  + Image (if available)
  + Question
      ↓
  Model Response

Turn 2
  Previous context
  + Feedback (if available)
  + Additional story information
  + Image (if available)
  + Question
      ↓
  Model Response

Turn 3
  ...

The model should retain relevant information from previous turns when answering subsequent questions.

Intended Use

CoMMET is intended for research on:

  • —Theory of Mind in LLMs and MLLMs
  • —social reasoning
  • —mental-state reasoning
  • —multi-turn reasoning
  • —multimodal reasoning
  • —perspective taking
  • —interactive evaluation of language and multimodal models

Citation

If you use CoMMET in your research, please cite:

bibtex
@misc{chen2026commetpsychologicallygroundedbenchmark,
      title={CoMMET: A Psychologically Grounded Benchmark for Evaluating Theory of Mind in Multimodal LLMs}, 
      author={Ruirui Chen and Weifeng Jiang and Chengwei Qin and Kaiwen Wei and Yanzhen Yue and Cheston Tan},
      year={2026},
      eprint={2603.11915},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2603.11915}, 
}

License

CoMMET is released under the MIT License.