vanyacohen/MET-Bench-Minecraft-Trajectories

MET-Bench: Multimodal Entity Tracking for Evaluating the Limitations of Vision-Language and Reasoning Models

Vanya Cohen and Raymond Mooney · ICML 2026

Paper · Evaluation code · Minecraft benchmark · Usage

Benchmark domains: Chess · Shell Game · Minecraft

Minecraft trajectories

This dataset contains the 462 source recordings used to construct the released MET-Bench Minecraft benchmark, comprising 462,235 captured observations. The recordings follow scripted behaviors for building, collecting wood, and exploring.

Each observation pairs an original first-person PNG screenshot with structured game telemetry and the recorded action. Telemetry includes player position, orientation, movement, controls, inventory, nearby blocks, and nearby entities. These parallel visual and structured representations support studying state changes throughout a recording.

The frames configuration contains one row per captured observation. The recordings configuration contains one metadata row per trajectory. Both retain the benchmark's trajectory assignments across train, validation, and test.

Dataset splits

Trajectories are disjoint across splits. Within each split, recordings are ordered by trajectory_id, and observations follow their original recording order. The configured capture interval is 200 milliseconds; recorded timestamps provide the actual timing.

The benchmark's trajectory_id identifies the source recording here. Its timestep indexes a processed trajectory, whereas frame_index here is the original screenshot number. Benchmark action instructions summarize transitions; action here preserves the collector's per-observation action label.

Usage

Install datasets and Pillow, then stream observations:

import json
from datasets import load_dataset

frames = load_dataset(
    "vanyacohen/MET-Bench-Minecraft-Trajectories",
    "frames",
    split="test",
    streaming=True,
)
frame = next(iter(frames))
image = frame["image"]  # PIL image
telemetry = json.loads(frame["telemetry"])
tick_record = json.loads(frame["tick_record"])

print(frame["trajectory_id"], frame["frame_index"], frame["action"])
print(telemetry["position"])

Load the compact recording index separately:

recordings = load_dataset(
    "vanyacohen/MET-Bench-Minecraft-Trajectories",
    "recordings",
    split="test",
)
print(recordings[0]["trajectory_id"], recordings[0]["num_frames"])

The images are embedded in Parquet shards grouped by split and trajectory. Streaming reads the dataset incrementally; the complete collection contains approximately 142 GB of source images and telemetry.

Frame fields

The telemetry capture timestamp and the tick record's timestamp come from their respective source records. Both are preserved in their original JSON fields.

Recording fields

num_frames is computed from the captured observations. Original metadata is preserved verbatim, including the zero frame count recorded for traj_collect_wood_20260321_195114, which contains 175 observations in this release.

Citation

@inproceedings{cohen2026metbench,
  title={MET-Bench: Multimodal Entity Tracking for Evaluating the Limitations of Vision-Language and Reasoning Models},
  author={Cohen, Vanya and Mooney, Raymond},
  booktitle={International Conference on Machine Learning},
  year={2026},
  url={https://arxiv.org/abs/2502.10886}
}

License

MIT.