open-world-agents/D2E-Original

D2E-Original

Project Page · Paper (arXiv) · GitHub · OWA Toolkit Documentation

This is the dataset for **D2E: Scaling Vision-Action Pretraining on Desktop Data for Transfer to Embodied AI**. 273.4 hours of synchronized video, audio, and input events from 29 PC games across diverse genres (FPS, open-world, sandbox, and more), for training vision-action models and game agents.

What's included:

• —Video + Audio: H.264 encoded at FHD/QHD 60fps with game audio.
• —Input events: Keyboard (press/release + key state), mouse (clicks, screen coordinates, raw HID deltas, button state), and active window info—all with nanosecond timestamps synchronized to video frames.
• —[OWAMcap](https://open-world-agents.github.io/open-world-agents/data/getting-started/why-owamcap/) format: Built on MCAP (widely adopted in robotics). Indexed for fast random access, crash-safe writes, and standardized message schemas that work across different datasets without custom parsing.

Recommended for: Training world model/video generation model. For training game agents with vision–action trajectories or pretraining vision–action models for transfer to embodied AI (e.g., robotic manipulation and navigation), we recommend using D2E-480p for efficient training at 480p resolution.

⚠️ 2026/01/14: We’ve completed fixes for issues introduced during reprocessing/re-filtering after applying privacy filtering (e.g., timestamps, MKV file paths). If you downloaded the data before this date, please re-download the updated version.

Visualize

Explore recordings directly in your browser with synchronized keyboard/mouse overlay: 👉 [Open in Dataset Visualizer](https://huggingface.co/spaces/open-world-agents/visualize_dataset?repo_id=open-world-agents/D2E-Original)

<img src="https://github.com/open-world-agents/owa-dataset-visualizer/blob/main/.github/assets/viewer.png?raw=true" alt="Dataset Visualizer Preview" width="600">

Load the data

Install mcap-owa-support (OWAMcap reader), owa-msgs (message type definitions), and huggingface_hub:

pip install mcap-owa-support owa-msgs huggingface_hub

Then load and iterate through the data:

from huggingface_hub import hf_hub_download
from mcap_owa.highlevel import OWAMcapReader

# Download a sample recording (mcap + video)
_kw = dict(repo_id="open-world-agents/D2E-480p", repo_type="dataset")
hf_hub_download(**_kw, filename="Apex_Legends/0805_01.mkv")
mcap_file = hf_hub_download(**_kw, filename="Apex_Legends/0805_01.mcap")

with OWAMcapReader(mcap_file) as reader:
    # Load a video frame
    for msg in reader.iter_messages(topics=["screen"]):
        screen = msg.decoded
        screen.resolve_relative_path(mcap_file)
        frame = screen.load_frame_array()  # numpy array (H, W, 3)
        break

    # Read keyboard events
    for msg in reader.iter_messages(topics=["keyboard"]):
        print(msg.decoded)  # KeyboardEvent(event_type='press', vk=87)
        break

    # Read raw mouse events
    for msg in reader.iter_messages(topics=["mouse/raw"]):
        print(msg.decoded)  # RawMouseEvent(last_x=12, last_y=-3, button_flags=0)
        break

Learn more: OWAMcap format guide

For training: We provide owa-data, a data pipeline that converts this dataset into HuggingFace Datasets ready for PyTorch DataLoader. It handles tokenization and sequence packing out of the box—so you can start training immediately without writing custom data loading code.

Structure

Each game folder contains paired .mcap + .mkv files:

Apex_Legends/
├── 0805_01.mcap    # Timestamped events + frame references
├── 0805_01.mkv     # Video + audio (FHD/QHD 60fps, H.264)
├── 0805_02.mcap
├── 0805_02.mkv
└── ...

The .mcap file stores lightweight MediaRef pointers to video frames instead of raw pixels—frames are decoded on-demand from the .mkv when you call load_frame_array(). MCAP files contain timestamped messages on these topics:

Games

Genres: FPS (Apex Legends, PUBG), open-world (Cyberpunk 2077, GTA V), simulation (Euro Truck Simulator 2), sandbox (Minecraft), roguelike (Brotato, Vampire Survivors), and more. 29 games released (273h) from 31 games collected (335h) after privacy filtering.

For 480p resolution, see D2E-480p.

Citation

@article{choi2025d2e,
    title={D2E: Scaling Vision-Action Pretraining on Desktop Data for Transfer to Embodied AI},
    author={Choi, Suhwan and Jung, Jaeyoon and Seong, Haebin and Kim, Minchan and Kim, Minyeong and Cho, Yongjun and Kim, Yoonshik and Park, Yubeen and Yu, Youngjae and Lee, Yunsung},
    journal={arXiv preprint arXiv:2510.05684},
    year={2025}
}