Cyn7hia-Z/MUSE

MUSE

Multimodal evaluation data.

Quick links: [[🌐 Website]](https://huggingface.co/datasets/Cyn7hia-Z/MUSE) [[📜 Paper]](https://arxiv.org/abs/2609.19088) [[💻 Code]](https://huggingface.co/datasets/Cyn7hia-Z/MUSE/tree/main/code)

Contents

1,800 test questions and 1,174 referenced images.

Affective Computing consists of four tasks: Object Classification, Emotion Detection, Visual Clue Identification, and Emotion Cause Inference. The multi-round prompt template covers these four tasks in sequence.

Splits and order

Only test data were supplied. data/train.jsonl is an empty placeholder and is excluded from the Hub configuration. Tasks are concatenated alphabetically by directory name. Question order within each source test_data.json is unchanged.

Dataset Format

The repository is organized as follows:

MUSE/
├── README.md
├── LICENSE
├── data/
│   ├── train.jsonl
│   ├── test.jsonl
│   └── test.parquet
├── images/
├── template/
└── code/generate_prompts.py

Each line in data/test.jsonl represents one image-question test case. Records share two fields and then provide annotations specific to their task:

{
  "image": "000001.png",       // Relative filename under images/
  "task": "Activity Localization", // One label from the taxonomy below
  "options": ["A. ...", "B. ..."], // Present for multiple-choice tasks
  "target": "B",               // Task-specific ground truth
  "bbx_normalized": [0.1, 0.2, 0.3, 0.4] // Present when grounding is required
}

The common fields are:

Other fields vary by task. Multiple-choice tasks generally use options and a letter-valued target; counting uses an integer target; Relative Position uses three directional targets; Remote Interaction uses interaction and evidence targets; and Affective Computing includes object, emotion, visual-evidence, and cause annotations. Bounding boxes follow COCO order [x, y, width, height]. Fields containing normalized use values in [0, 1], while fields containing abs use image pixels. Fields without either suffix retain the source scale.

The distributed files have these roles:

• —data/test.jsonl: original fields and types, with task added and image replaced by the sequential filename. Resolve it relative to MUSE/images/.
• —data/test.parquet: the same questions, in the same order, with task, image (embedded bytes and path), image_name (sequential filename), and annotation (JSON string containing all other original fields). Mixed scalar types and heterogeneous bounding-box lists require this lossless JSON representation. Parse annotation with json.loads to recover its fields.
• —images/: referenced image bytes only, renamed with six-digit sequential numbers and preserving file extensions. Numbers follow sorted source basenames.
• —template/: task prompt templates; the superseded emotion.txt is excluded.
• —code/generate_prompts.py: generate prompts or image-message payloads for every JSONL entry using the packaged templates.

The Parquet file uses a stable cross-task schema because the JSONL annotations are heterogeneous:

Tag Taxonomy

The task field is the primary per-example tag. Its nine values are grouped by the capability they evaluate:

• —Visual grounding: Activity Localization selects the bounding box that grounds an activity, and Activity Description selects the description for a grounded region.
• —Recognition: Object Count, Scene Classification, and Culture Identification evaluate object quantity, scene type, and culturally relevant visual content.
• —Spatial and relational reasoning: Relative Position predicts lateral, depth, and vertical relations; Remote Interaction identifies an interaction target and its visual evidence.
• —Compositional reasoning: Jigsaw Puzzle selects the missing image region.
• —Affective understanding: Affective Computing covers four linked stages: Object Classification, Emotion Detection, Visual Clue Identification, and Emotion Cause Inference.

The tags in the dataset-card YAML header are repository-level discovery tags; they are not additional per-example annotations. Use task to group or report benchmark results.

Loading the JSONL files

Each line in data/test.jsonl is one complete JSON object. The records retain the task-specific fields from the source data and add task. The image value is a filename relative to the dataset's images/ directory.

Use Python's standard library when you want the records exactly as stored:

import json
from pathlib import Path

dataset_root = Path("data/MUSE")
with (dataset_root / "data/test.jsonl").open(encoding="utf-8") as file:
    test_data = [json.loads(line) for line in file if line.strip()]

example = test_data[0]
image_path = dataset_root / "images" / example["image"]
print(example["task"], image_path)

For memory-efficient iteration, read one line at a time instead of constructing the list:

with (dataset_root / "data/test.jsonl").open(encoding="utf-8") as file:
    for line in file:
        if not line.strip():
            continue
        example = json.loads(line)
        image_path = dataset_root / "images" / example["image"]
        # Run inference for this example here.

The tasks contain heterogeneous nested fields, including bounding boxes with different shapes and value types. For that reason, do not load the complete JSONL file directly with datasets.load_dataset("json", ...), which requires a single Arrow-compatible schema. Use the standard-library examples above for JSONL, or use the Parquet representation below with Hugging Face Datasets.

data/train.jsonl is intentionally empty because no training split was supplied.

Loading the Parquet file

import json
from datasets import Features, load_dataset
import pyarrow.parquet as pq

path = "data/MUSE/data/test.parquet"
features = Features.from_arrow_schema(pq.read_schema(path))
ds = load_dataset("parquet", data_files={"test": path}, features=features,
                  split="test", streaming=True, batch_size=32)
example = next(iter(ds))
image = example["image"]
annotation = json.loads(example["annotation"])
# After uploading the contents of MUSE to a dataset repository:
# ds = load_dataset("OWNER/MUSE", batch_size=32)

Use small read batches with older PyArrow versions to keep embedded image data below their per-batch binary size limit. The default Hub configuration selects only Parquet to avoid loading the JSONL copy as duplicate questions. Images are embedded for portable loading. See Hugging Face's image format documentation and configuration documentation.

Generating prompts

code/generate_prompts.py reads each JSONL record, selects the matching file in template/, fills its placeholders, and writes a new JSONL record containing two additional fields:

• —prompt: a string for regular tasks or a four-item list for Affective Computing.
• —messages: image-and-text message payloads suitable for a multimodal chat API.

From the MUSE/ directory, generate prompts for the full test split with:

python code/generate_prompts.py --output data/test_with_prompts.jsonl

To place a URL prefix in each generated image message, pass --image-base:

python code/generate_prompts.py \
  --input data/test.jsonl \
  --output data/test_with_prompts.jsonl \
  --image-base https://huggingface.co/datasets/OWNER/MUSE/resolve/main/images

Without --output, the generated records are printed to standard output. You can also call the generator from Python:

import json
import sys
from pathlib import Path

dataset_root = Path("data/MUSE")
sys.path.insert(0, str(dataset_root / "code"))
from generate_prompts import generate_messages, generate_prompt

with (dataset_root / "data/test.jsonl").open(encoding="utf-8") as file:
    entry = json.loads(next(file))

prompt = generate_prompt(entry, dataset_root / "template")
messages = generate_messages(entry, dataset_root / "template")

Reproduction

From the source project, install huggingface/requirements.txt and run python huggingface/prepare_dataset.py --output data/MUSE --overwrite. Source data are never modified.

Team

MUSE was created by:

• —Luyao Zhu — AI Singapore
• —Xun Wei Yee — AI Singpore
• —Wesley Tay Li Wen - AI Singapore
• —Qum Lim — AI Singapore

with technical support by:

• —Mak Mun Thye — AI Singapore
• —Kenneth Yau Weng Kuan — AI Singpore
• —Chin Zhi Qi, Joel — AI Singapore

MUSE is maintained by:

• —Luyao Zhu — AI Singapore

Contact

For questions about the dataset, annotations, evaluation, or permitted use, contact Luyao Zhu at luyaozhu@outlook.com. You may also open a discussion in the MUSE dataset repository.

Citation

If you use MUSE in your research, please cite:

@misc{zhu2026musebenchmarkinglargevisionlanguage,
      title={MUSE: Benchmarking Large Vision-Language Models on Multi-Modal Understanding in Situated Education}, 
      author={Luyao Zhu and Xun Wei Yee and Wei Li and Mun Thye Mak and Wee Siong Ng},
      year={2026},
      eprint={2609.19088},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2609.19088}, 
}

Copyright Statement

The MUSE annotations, prompt templates, data-packaging code, and repository metadata are released under the MIT License; see LICENSE for the full terms. Copyright in third-party images and any depicted logos, trademarks, artworks, or other protected material remains with the respective rights holders. The MIT License does not grant additional rights to that third-party content. Users are responsible for ensuring that their use complies with applicable licenses, copyright, privacy, and personality-rights requirements.

For copyright questions, attribution corrections, or removal requests, contact the person listed above or open a discussion in the MUSE dataset repository on Hugging Face.