tsakman23/visual_masked_distracting_metaworld

Visual Masked Distracting Meta-World (ground-truth masks)

• —Author: Georgios Tsakoumakis
• —Thesis: Interaction-Masked Latent Action Models for Object-Aware Manipulation under Visual Distractors (MSc, Imperial College London)

Expert Meta-World manipulation trajectories rendered with dynamic video-background distractors, augmented with ground-truth segmentation masks and pose for the manipulated object: the agent mask plus two per-frame fields, object_mask and object_state.

All masks in this repository are ground truth, read directly from the MuJoCo simulator's segmentation rendering at data-generation time - they are not the output of a learned segmenter (e.g. SAM). This makes the dataset suitable as a clean supervision or evaluation signal for object-centric world models and probes.

Why this dataset

Distracting Meta-World places a task-irrelevant, moving video background behind the scene, so pixel reconstruction alone no longer isolates the controllable content. The agent mask identifies the robot; the added object mask identifies the manipulated object. Together they let a model (or an evaluator) separate embodiment, manipulated object, and distractor background. The dataset was built to support:

• —IM-LAM - a directed agent to object forward-dynamics model over agent + object masks.
• —Foreground-MaskLAM - gating the world-model reconstruction loss by the agent-and-object union so the decoder must reconstruct the manipulated object, not just the arm.

Data fields

Every example is a single timestep t of an expert trajectory:

(observation, action, reward, terminated, truncated) follow the standard (o_t, a_t, r_t, term_t, trunc_t) convention. Frames are stored in trajectory order, so a temporal stack of length k can be formed by slicing consecutive rows.

Configurations (tasks)

The dataset is organized as one HuggingFace builder config per Meta-World task. Load a task by passing its bare config name (no masked-/distracting- prefix), for example push-v3 or handle-pull-v3. Available task configs include:

push-v3         sweep-into-v3       door-open-v3    handle-pull-v3
pick-place-v3   peg-insert-side-v3  dial-turn-v3

Splits

Per task configuration:

Usage

from datasets import load_dataset

ds = load_dataset(
    "tsakman23/visual_masked_distracting_metaworld",
    name="handle-pull-v3",   # bare task config name
    split="train",
)

ex = ds[0]
ex["observation_distracted"]  # PIL.Image, 128x128 RGB (distracted frame)
ex["mask"]                    # PIL.Image, 128x128 L, agent = 255
ex["object_mask"]             # PIL.Image, 128x128 L, manipulated object = 255
ex["object_state"]            # [x, y, z, qw, qx, qy, qz]
ex["action"]                  # [dx, dy, dz, gripper]

To load only the columns you need (masks are large), pass columns=... when constructing a torch-formatted view, e.g. ["observation_distracted", "mask", "object_mask", "object_state", "action"].

For distracting/masked training you typically use observation_distracted as the observation and drop the vanilla observation column.

How the masks were produced

Masks are the simulator's own ground-truth segmentation, obtained from MuJoCo's segmentation render (per-geom object IDs) at the same camera pose as the RGB frame, then binarized into the agent mask (mask) and the manipulated-object mask (object_mask). object_state is the manipulated object's body pose read from the simulator. No learned segmentation model is involved, so the masks are exact up to rendering resolution.

Masks

This repository contains only ground-truth masks - the simulator's agent mask and object_mask. There are no predicted (pred_mask / pred_object_mask) columns.

Licensing and attribution

• —Meta-World environments and scripted expert policies: MIT License (Yu et al., "Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning", CoRL 2019).
• —Distractor backgrounds are video frames from the DAVIS dataset, which is released for non-commercial research use; downstream use of the distracted frames should respect DAVIS's terms.

This dataset is released under CC BY 4.0 (attribution required). One carve-out: because the observation_distracted frames embed DAVIS video content, that portion is subject to DAVIS's research-only terms - review the DAVIS license before any commercial use or redistribution of the distracted frames. The vanilla observation, masks, object_state, state, and action fields derive from Meta-World (MIT) and its scripted experts.

Citation

This dataset was produced as part of the following MSc thesis:

Georgios Tsakoumakis. Interaction-Masked Latent Action Models for Object-Aware Manipulation under Visual Distractors. MSc thesis, Imperial College London, 2026.

@mastersthesis{tsakoumakis2026imlam,
  title  = {Interaction-Masked Latent Action Models for Object-Aware Manipulation under Visual Distractors},
  author = {Tsakoumakis, Georgios},
  school = {Imperial College London},
  year   = {2026},
  type   = {{MSc} thesis}
}

If you use this dataset, please also cite Meta-World and the DAVIS background dataset.