Datasetpublic

Kiva12138/mirth_lerobot

MIRTH Dataset Multi-camera real-world manipulation demonstrations for history-aware Vision-Language-Action agents The MIRTH dataset is a real-world robot manipulation dataset collected on a physical LeRobot platform. It contains synchronized main-camera and wrist-camera observations, robot proprioception, language instructions, and expert action trajectories for training and evaluating Vision-Language-Action (VLA) agents. This release provides the same… See the full description on the dataset page: https://huggingface.co/datasets/Kiva12138/mirth_lerobot.

sourceHugging Faceotherupdated 3mo agoView on Hugging Face

0likes150downloads

Dataset Card

MIRTH Dataset

Multi-camera real-world manipulation demonstrations for history-aware Vision-Language-Action agents

![Paper](https://aclanthology.org/2026.acl-long.1016/) ![arXiv](https://arxiv.org/abs/2606.31167) ![Code](https://github.com/kiva12138/MIRTH)

</div>

The MIRTH dataset is a real-world robot manipulation dataset collected on a physical LeRobot platform. It contains synchronized main-camera and wrist-camera observations, robot proprioception, language instructions, and expert action trajectories for training and evaluating Vision-Language-Action (VLA) agents.

This release provides the same demonstrations in two formats:

Format	Hugging Face dataset repo	Recommended use
LeRobot	`Kiva12138/mirth_lerobot`	Compatibility with LeRobot tooling and inspection workflows.
RLDS / TFDS	`Kiva12138/mirth_rlds`	Recommended format for MIRTH training and evaluation code.

Dataset Summary

MIRTH is organized into five levels of increasing semantic and control complexity. Each level contains four tasks, and each task contains 50 expert demonstration episodes, for a total of 20 tasks and 1000 episodes.

Level	Focus	Tasks	Episodes per task	Episodes
Basic manipulation	Atomic pick-and-place and target placement skills	4	50	200
Mechanism operation	Drawer opening / closing and object insertion with articulated mechanisms	4	50	200
Scene rearrangement	Multi-object workspace organization and spatial rearrangement	4	50	200
Category reasoning	Object grouping by category, color, attribute, or exclusion constraints	4	50	200
Recipe-level semantic composition	Long-horizon semantic tasks requiring high-level instruction grounding	4	50	200
Total		20		1000

Demonstrations were collected under randomized object poses and workspace configurations to support robust imitation learning and evaluation.

Visual Examples

The examples below show two demonstration episodes from the synchronized main and wrist cameras.

<table align="center"> <tr> <td align="center"><strong>Sample 1: main camera</strong></td> <td align="center"><strong>Sample 1: wrist camera</strong></td> <td align="center"><strong>Sample 2: main camera</strong></td> <td align="center"><strong>Sample 2: wrist camera</strong></td> </tr> <tr> <td><a href="https://raw.githubusercontent.com/kiva12138/MIRTH/main/assets/m1.mp4"><img src="https://raw.githubusercontent.com/kiva12138/MIRTH/main/assets/m1preview.gif" alt="Sample 1 main camera preview" width="100%"></a></td> <td><a href="https://raw.githubusercontent.com/kiva12138/MIRTH/main/assets/w1.mp4"><img src="https://raw.githubusercontent.com/kiva12138/MIRTH/main/assets/w1preview.gif" alt="Sample 1 wrist camera preview" width="100%"></a></td> <td><a href="https://raw.githubusercontent.com/kiva12138/MIRTH/main/assets/m2.mp4"><img src="https://raw.githubusercontent.com/kiva12138/MIRTH/main/assets/m2preview.gif" alt="Sample 2 main camera preview" width="100%"></a></td> <td><a href="https://raw.githubusercontent.com/kiva12138/MIRTH/main/assets/w2.mp4"><img src="https://raw.githubusercontent.com/kiva12138/MIRTH/main/assets/w2preview.gif" alt="Sample 2 wrist camera preview" width="100%"></a></td> </tr> </table>

Tasks

ID	Task
task1	Place the banana in the plate on the right
task2	Place the brown kiwi on the cutting board
task3	Place the carrot in the plate on the left
task4	Place the star fruit in the white frying pan
task5	Clear the cooking area move all food items off the cutting board and leave only tools on the cutting board
task6	Move any fruits that are directly on the table into the pot with the dark lid and any vegetables that are directly on the table into the white frying pan
task7	Place all vegetables except the corn with green leaves into the pot with the dark lid
task8	Put all fruits except the banana into the white frying pan
task9	Close the second drawer of the four-drawer cabinet
task10	Open the second drawer, put the banana into it, and close the drawer
task11	Open the top drawer of the four-drawer cabinet
task12	Open the top drawer, place the spatula inside it, and close the drawer
task13	Clean up the workspace by moving all fruits onto the left white plate and all vegetables onto the right white plate
task14	Clear the white frying pan by moving any items inside it onto the cutting board, leaving the frying pan empty
task15	Empty the small bucket onto the cutting board
task16	Swap all items currently on the left white plate with the items on the right white plate
task17	Prepare a breakfast plate by placing the cooked fried egg, one fruit, and one vegetable together on the right white plate
task18	Prepare ingredients for a fruit yogurt by placing the strawberry, kiwi, apple pieces, and banana into the white frying pan
task19	Prepare ingredients for a simple vegetable scramble by placing the raw egg, carrot, green bean, and yellow bell pepper onto the cutting board, and leave all fruits where they are
task20	Put all fruits that are good for a refreshing snack, the orange, kiwi, strawberry, and star fruit, on the left white plate

Repository Layout

LeRobot Format

The LeRobot release preserves the semantic grouping used during collection:

text

Kitchen_LeRobot/
  Basic Tasks/
  Category Reasoning/
  Mechanism Operations/
  Scene Rearrange/
  Semantic Recipe/

Each task folder contains the standard LeRobot-style data/, meta/, and videos/ subdirectories.

RLDS / TFDS Format

The RLDS release stores one TFDS-style dataset per task:

text

Kitchen_RLDS/
  task1/
    1.0.0/
  task2/
    1.0.0/
  ...
  task20/
    1.0.0/
  task_metadata.json

task_metadata.json maps the compact dataset names (task1 ... task20) to the original natural-language task names.

Loading

The MIRTH codebase provides loaders for both dataset formats:

—rlds_datasets.RLDSDataset for RLDS / TFDS data.
—lerobot_datasets.LeRobotOpenVLADataset for LeRobot data.

The RLDS / TFDS format is the recommended training format for the MIRTH codebase. Both loaders adapt samples to the same OpenVLA-style batch contract before collation, so they can share PaddedCollatorForActionPrediction.

Example RLDS-style usage in the MIRTH repository:

python

from rlds_datasets import RLDSBatchTransform, RLDSDataset
from utils.data_utils import PaddedCollatorForActionPrediction

See the project repository for full training and smoke-test scripts:

—Code: https://github.com/kiva12138/MIRTH
—RLDS loader utilities: rlds_datasets/
—LeRobot loader utilities: lerobot_datasets/
—Converter: lerobot_to_rlds.py
—Smoke tests: TestDataset.py and TestLeRobotDataset.py

Paper

MIRTH is introduced in:

MIRTH: Mutual-Information Reasoning with Temporal Hubs for Vision-Language-Action Agents Hao Sun, Yu Song, Shiyu Teng, Ziwei Niu, Yen-Wei Chen ACL 2026 Long Papers

Please refer to the arXiv version for corrected notation:

—ACL Anthology: https://aclanthology.org/2026.acl-long.1016/
—arXiv: https://arxiv.org/abs/2606.31167
—Code: https://github.com/kiva12138/MIRTH

Citation

If you use MIRTH or the MIRTH dataset, please cite:

bibtex

@inproceedings{sun-etal-2026-mirth,
    title = "{MIRTH}: Mutual-Information Reasoning with Temporal Hubs for Vision-Language-Action Agents",
    author = "Sun, Hao and
      Song, Yu and
      Teng, Shiyu and
      Niu, Ziwei and
      Chen, Yen-Wei",
    editor = "Liakata, Maria and
      Moreira, Viviane P. and
      Zhang, Jiajun and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.acl-long.1016/",
    pages = "22199--22215",
    ISBN = "979-8-89176-390-6"
}

License and Contact

The dataset is released for research use. Please check the repository and paper for the latest license and usage notes.

For questions about the paper or released resources, contact Hao Sun: sunhaoxx@zju.edu.cn.