vanyacohen/MET-Bench-Minecraft

MET-Bench: Multimodal Entity Tracking for Evaluating the Limitations of Vision-Language and Reasoning Models

Vanya Cohen and Raymond Mooney · ICML 2026

Paper · Publication page · Load the dataset · Citation

Domains: Chess · Shell Game · Minecraft

MET-Bench evaluates entity state tracking across text and image modalities. This repository contains the Minecraft domain.

Minecraft

Minecraft is a state prediction task involving partial observations, dynamic environments, and visually complex scenes. Given an initial player state and an action instruction, the model selects the correct next state from four candidates.

Examples are drawn from recorded trajectories of scripted behaviors: collecting wood, exploring, and building structures. Actions describe player inputs such as walking, turning, attacking, using an item, climbing, and descending. One instruction can combine several inputs, such as “Walk forward 8 blocks, attack.”

Parallel text and image modalities

Each example contains two aligned representations of its initial state and four candidate next states:

Text states contain structured game telemetry, including player position, orientation, velocity, health, food, hotbar contents and held item when available, nearby blocks, and nearby entities. Image states are corresponding first-person screenshots.

The entries candidate_states[i] and image_candidate_states[i] describe the same candidate observation. Both representations share the same action and correct answer.

The paper evaluates text-state and image-state inputs with text actions. The dataset also includes image_action, a rendered diagram of the action instruction.

Dataset splits

Source trajectories are disjoint across training, validation, and test. Each example contains one initial state, one action instruction, and four candidate next states.

The paper reports Minecraft results on a 500-example subset of the test split. This release includes the complete splits.

Evaluation subset

The evaluation configuration contains a single 500-example test split for benchmark evaluation. Examples are the first 500 unique next-state tasks in the full test split, in source order. Uniqueness is determined by the initial state, action instruction, and ordered candidate states. The initial state, four candidates, images, and correct answer are preserved.

The evaluation_text_only configuration contains the same examples in the same order, with image fields omitted for text-only evaluation. Both configurations preserve the source example_id values and all text fields. The full configuration remains the default.

from datasets import load_dataset

test = load_dataset("vanyacohen/MET-Bench-Minecraft", "evaluation", split="test")
text_test = load_dataset("vanyacohen/MET-Bench-Minecraft", "evaluation_text_only", split="test")

Evaluate with lmms-eval

Run the complete MET-Bench evaluation across Minecraft, Chess, and Shell Game, with separate text and image tasks, using lmms-eval. Each task evaluates 500 examples. Chess and Shell Game use ten-action sequences; Minecraft uses next-state prediction tasks.

Install the current version from GitHub:

python -m pip install -U git+https://github.com/EvolvingLMMs-Lab/lmms-eval.git

For an OpenAI-compatible API, set OPENAI_API_KEY in your environment and run:

python -m lmms_eval \
  --model openai \
  --model_args model_version=gpt-4o-mini \
  --tasks metbench \
  --batch_size 1 \
  --log_samples \
  --output_path results/metbench

Replace gpt-4o-mini with your model ID. For another compatible provider, also set OPENAI_API_BASE. The complete evaluation requires a model that supports text and multiple images. Results and sample outputs are saved to results/metbench.

See the MET-Bench task documentation for task names and additional options.

Usage

import json
from datasets import load_dataset

dataset = load_dataset(
    "vanyacohen/MET-Bench-Minecraft",
    "full",
    streaming=True,
)
example = next(iter(dataset["test"]))

# Text states
initial_state = json.loads(example["initial_state"])
candidate_states = [
    json.loads(state) for state in example["candidate_states"]
]

# Corresponding images, decoded as PIL images
image_initial_state = example["image_initial_state"]
image_candidate_states = example["image_candidate_states"]

action = example["action"]
target_index = example["correct_choice"] - 1

target_state = candidate_states[target_index]
target_image = image_candidate_states[target_index]

correct_choice numbers the four candidates from 1 to 4. Subtracting one gives the corresponding Python list index.

Data fields

The dataset is stored in Parquet shards with text and embedded images together in each row.

The correct candidate is the recorded state after the action. The other three candidates are drawn from other points in the same trajectory.

Citation

@inproceedings{cohen2026metbench,
  title={MET-Bench: Multimodal Entity Tracking for Evaluating the Limitations of Vision-Language and Reasoning Models},
  author={Cohen, Vanya and Mooney, Raymond},
  booktitle={International Conference on Machine Learning},
  year={2026},
  url={https://arxiv.org/abs/2502.10886}
}

License

MIT.