Datasetpublic

tencent/Hy-Embodied-0.5-VLA-Data

Hy-Embodied-0.5-VLA From Vision-Language-Action Models to a Real-World Robot Learning Stack Tencent Robotics X × Tencent Hy Team 📖 Abstract We introduce Hy-Embodied-0.5-VLA (Hy-VLA) — an end-to-end Vision-Language-Action system that spans the full robot learning stack: data collection, model design, pre-training, supervised fine-tuning, RL post-training, and real-world deployment. Built on the Hy-Embodied-0.5 MoT backbone, Hy-VLA integrates a flow-matching… See the full description on the dataset page: https://huggingface.co/datasets/tencent/Hy-Embodied-0.5-VLA-Data.

sourceHugging Facecc-by-4.0updated 3mo agoView on Hugging Face

23likes81kdownloads

Dataset Card

<div align="center"> <h1>Hy-Embodied-0.5-VLA</h1> <p><b>From Vision-Language-Action Models to a Real-World Robot Learning Stack</b></p> <p><i>Tencent Robotics X × Tencent Hy Team</i></p>

📖 Abstract

We introduce Hy-Embodied-0.5-VLA (Hy-VLA) — an end-to-end Vision-Language-Action system that spans the full robot learning stack: data collection, model design, pre-training, supervised fine-tuning, RL post-training, and real-world deployment. Built on the Hy-Embodied-0.5 MoT backbone, Hy-VLA integrates a flow-matching action expert, a compact memory encoder for multi-frame history, and a delta-chunk action representation decoupled from embodiment-specific kinematics.

Powered by 10,000+ hours of high-fidelity UMI demonstrations collected via a custom fingertip interface with optical motion-capture, Hy-VLA achieves state-of-the-art results on the RoboTwin 2.0 benchmark (90.9% / 90.1% on Clean / Randomized) and demonstrates robust cross-embodiment transfer across four real-world robot platforms. Paired with FlowPRO preference optimization and an asynchronous inference framework, Hy-VLA establishes a scalable paradigm for continuous dexterous manipulation.

Overview

Hy-Embodied-0.5-VLA-Data is a large-scale bimanual manipulation dataset for training Vision-Language-Action (VLA) foundation models. Powered by 2000+ hours of high-fidelity demonstrations collected via a custom fingertip UMI device with optical motion-capture, it spans 70+ manipulation tasks. The dataset is released in Lance format compatible with LeRobot v3.0.

Note: The open-source release contains approximately 20% of the full corpus.

Dataset Statistics

Property	Value
Total Episodes	250,304
Total Frames	233,600,314
Total Duration	2,163 hours
Total Size	~18.8 TB (22 tables)
Frequency	30 Hz
Cameras	3 views (head + left wrist + right wrist)
Resolution	240 × 424 px per camera
Format	Lance (LeRobot v3.0 schema)
Tables	22 (table000 ~ table021, ~100h each)

Directory Structure

Each table is a self-contained LeRobot v3.0 dataset root:

table_000/
├── table_000.lance/          # Lance columnar data (5GB shards)
│   ├── _versions/
│   └── data/
│       └── ...-data-0.lance
└── meta/                     # LeRobot v3.0 metadata
    ├── info.json             # Table-level summary
    ├── stats.json            # Per-feature statistics
    ├── tasks.parquet         # Task ↔ index mapping
    └── episodes/             # Episode boundary parquet files

Data Schema

Each row (frame) contains:

Observations

Column	Type	Shape	Description
`observation.state`	`float32`	`[16]`	Dual-arm EEF state: left [x,y,z,qx,qy,qz,qw,gripper], right [x,y,z,qx,qy,qz,qw,gripper]
`observation.images.cam_high`	`image`	`[240,424,3]`	Overhead camera RGB image
`observation.images.cam_left_wrist`	`image`	`[240,424,3]`	Left wrist-mounted camera RGB image
`observation.images.cam_right_wrist`	`image`	`[240,424,3]`	Right wrist-mounted camera RGB image

Actions

Column	Type	Shape	Description
`action`	`float32`	`[2]`	Gripper openness [left, right] derived from state

Metadata

Column	Type	Shape	Description
`task_index`	`int32`	`[1]`	Task ID mapping to `meta/tasks.parquet`
`task`	`string`	`[1]`	Task description (Chinese, e.g., "抓取红色方块并放入盒子")
`episode_index`	`int32`	`[1]`	Global unique episode index
`frame_index`	`int32`	`[1]`	Frame index within episode (starts at 0)
`timestamp`	`float32`	`[1]`	Seconds from episode start

Usage

LanceTableReader reads a single Lance table (local or HF Hub):

python

from hy_vla.data.lance_dataset import LanceTableReader

# Local directory
reader = LanceTableReader(root="./table_000")

# HF Hub
reader = LanceTableReader(
    repo_id="tencent/Hy-Embodied-0.5-VLA-Data",
    table_name="table_000",
)

# Access
frame = reader[42]                        # single frame dict
episode = reader.get_episode(3)           # all frames of episode 3

Also compatible with raw lance, lancedb, and `lerobot-lancedb` (LeRobotLanceDataset).

Episode Visualization

bash

# Use the HF Hub dataset, pick table_000 episode 666
python scripts/vis_umi_episode.py -t table_000 -e 666

# Local Lance root
python scripts/vis_umi_episode.py /path/to/Hy-Embodied-0.5-Data -e 0 --no-3d

Downloading Specific Tables

Due to the large total size (~18.8 TB), you may prefer to download individual tables:

python

from huggingface_hub import snapshot_download

# Download only table_000 (~890 GB)
snapshot_download(
    "tencent/Hy-Embodied-0.5-VLA-Data",
    allow_patterns="table_000/**",
    repo_type="dataset"
)

📚 Citation

If you find Hy-VLA useful for your research, please cite:

bibtex

@article{zhang2026hy,
  title={Hy-Embodied-0.5-VLA: From Vision-Language-Action Models to a Real-World Robot Learning Stack},
  author={Zhang, He and Xiang, Lingzhu and Lin, Haitao and Huang, Zeyu and Wang, Minghui and Zhong, Dingyan and Dong, Yubo and Wu, Yihao and Rao, Yongming and Zhang, Dongsheng and others},
  journal={arXiv preprint arXiv:2606.14409},
  year={2026}
}

License

This dataset is released under CC-BY-4.0.