datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
Challenge-phase1-dataset
Post-training for Robotics Foundation Models — Challenge Phase 1 Dataset
This is the public Phase 1 dataset for the RSS 2026 Workshop & Challenge on Post-training for Robotics Foundation Models.
Workshop website: https://posttraining-for-robotics.github.io/
The dataset contains real-robot bimanual manipulation trajectories on three benchmark tasks, collected on a bimanual YAM follower teleoperated by a GELLO leader arm. Every frame is timestamp-aligned across joint state, action… See the full description on the dataset page: https://huggingface.co/datasets/Posttraining-RFM-RSS2026/Challenge-phase1-dataset.store_play_v2_en_phase6_roistore_play_v2_en_phase4_roiphase1_sort_by_color_A1_10fps
Phase1 Sort By Color A1 10Fps
LeRobot v3.0 dataset collected via
SCRAPE-IsaacLab — a
Code-as-Policies replay pipeline running inside Isaac Sim 5.1 / IsaacLab 2.3.2.
Task instruction: "Sort the blocks onto the matching colored dishes."
Robot: so101_follower
Cameras: top + left-wrist RGB @ 10 fps
Episodes: 100 (74,322 frames total)
Labels: per-frame natural-language skill labels in skill.natural_language
and subtask.* columns (labeled by Gemini)
Generated on Isaac Sim 5.1 /… See the full description on the dataset page: https://huggingface.co/datasets/HyeonseokE/phase1_sort_by_color_A1_10fps.store_play_v2_en_phase3phase1_sort_by_color_A2_10fps
Phase1 Sort By Color A2 10Fps
LeRobot v3.0 dataset collected via
SCRAPE-IsaacLab — a
Code-as-Policies replay pipeline running inside Isaac Sim 5.1 / IsaacLab 2.3.2.
Task instruction: "Sort the blocks onto the matching colored dishes."
Robot: so101_follower
Cameras: top + left-wrist RGB @ 10 fps
Episodes: 100 (74,921 frames total)
Labels: per-frame natural-language skill labels in skill.natural_language
and subtask.* columns (labeled by Gemini)
Generated on Isaac Sim 5.1 /… See the full description on the dataset page: https://huggingface.co/datasets/HyeonseokE/phase1_sort_by_color_A2_10fps.store_play_v2_en_phase5_roiturn_on_lever_ours_phase1_80_10fps
Turn On Lever Ours Phase1 80 10Fps
LeRobot v3.0 dataset collected via
SCRAPE-IsaacLab — a
Code-as-Policies replay pipeline running inside Isaac Sim 5.1 / IsaacLab 2.3.2.
Task instruction: "Turn the lever on; the status indicator should turn green."
Robot: so101_follower
Cameras: top + left-wrist RGB @ 10 fps
Episodes: 80 (17,457 frames total)
Labels: per-frame natural-language skill labels in skill.natural_language
and subtask.* columns (labeled by Gemini)
Generated on Isaac… See the full description on the dataset page: https://huggingface.co/datasets/HyeonseokE/turn_on_lever_ours_phase1_80_10fps.phase1_sort_by_color_A2_10fps_via4cm
Phase1 Sort By Color A2 10Fps Via4Cm
LeRobot v3.0 dataset collected via
SCRAPE-IsaacLab — a
Code-as-Policies replay pipeline running inside Isaac Sim 5.1 / IsaacLab 2.3.2.
Task instruction: "Sort the blocks onto the matching colored dishes."
Robot: so101_follower
Cameras: top + left-wrist RGB @ 10 fps
Episodes: 100 (74,827 frames total)
Labels: per-frame natural-language skill labels in skill.natural_language
and subtask.* columns (labeled by Gemini)
Generated on Isaac Sim 5.1… See the full description on the dataset page: https://huggingface.co/datasets/HyeonseokE/phase1_sort_by_color_A2_10fps_via4cm.close_box_ours_phase1_80_10fps
Close Box Ours Phase1 80 10Fps
LeRobot v3.0 dataset collected via
SCRAPE-IsaacLab — a
Code-as-Policies replay pipeline running inside Isaac Sim 5.1 / IsaacLab 2.3.2.
Task instruction: "Close the box by placing the lid on the box body."
Robot: so101_follower
Cameras: top + left-wrist RGB @ 10 fps
Episodes: 80 (22,566 frames total)
Labels: per-frame natural-language skill labels in skill.natural_language
and subtask.* columns (labeled by Gemini)
Generated on Isaac Sim 5.1 /… See the full description on the dataset page: https://huggingface.co/datasets/HyeonseokE/close_box_ours_phase1_80_10fps.xlerobot-desk-marker-phase1
saivishwak/xlerobot-desk-marker-phase1
This dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v3.0",
"fps": 30,
"features": {
"action": {
"dtype": "float32",
"shape": [
12
],
"names": [
"left_arm_shoulder_pan.pos",
"left_arm_shoulder_lift.pos",
"left_arm_elbow_flex.pos",
"left_arm_wrist_flex.pos",
"left_arm_wrist_roll.pos"… See the full description on the dataset page: https://huggingface.co/datasets/saivishwak/xlerobot-desk-marker-phase1.g1-pipette-tip-teleop-phase
g1-pipette-tip-teleop-phase
Unitree G1 + Inspire RH56 teleoperation: attach a red pipette tip from a rack to
a pipette, then eject it into a disposal box. LeRobot v2.1,
73 episodes / 68,585 frames @ 60 fps.
Two things distinguish this from the earlier releases: a detector-tracked zoom
stream on the tip rack, and per-frame phase sentences as a second language
channel.
Streams
key
content
observation.images.rgb
1280x720 ego view, re-encoded with a 2 px… See the full description on the dataset page: https://huggingface.co/datasets/jren313/g1-pipette-tip-teleop-phase.open_box_ours_phase1_80_10fps
Open Box Ours Phase1 80 10Fps
LeRobot v3.0 dataset collected via
SCRAPE-IsaacLab — a
Code-as-Policies replay pipeline running inside Isaac Sim 5.1 / IsaacLab 2.3.2.
Task instruction: "Open the box by moving the lid to the target marker."
Robot: so101_follower
Cameras: top + left-wrist RGB @ 10 fps
Episodes: 80 (23,214 frames total)
Labels: per-frame natural-language skill labels in skill.natural_language
and subtask.* columns (labeled by Gemini)
Generated on Isaac Sim 5.1 /… See the full description on the dataset page: https://huggingface.co/datasets/HyeonseokE/open_box_ours_phase1_80_10fps.turn_off_lever_ours_phase1_80_10fps
Turn Off Lever Ours Phase1 80 10Fps
LeRobot v3.0 dataset collected via
SCRAPE-IsaacLab — a
Code-as-Policies replay pipeline running inside Isaac Sim 5.1 / IsaacLab 2.3.2.
Task instruction: "Turn the lever off; the status indicator should turn red."
Robot: so101_follower
Cameras: top + left-wrist RGB @ 10 fps
Episodes: 80 (17,232 frames total)
Labels: per-frame natural-language skill labels in skill.natural_language
and subtask.* columns (labeled by Gemini)
Generated on Isaac… See the full description on the dataset page: https://huggingface.co/datasets/HyeonseokE/turn_off_lever_ours_phase1_80_10fps.onevision-phase2-data
OneVision-Encoder Phase 2 — caches (data)
Pre-computed feature caches required to train Phase 2 (video generation) experiments on top of the a8 codec.
Companion repo: code + frozen a8 ckpt live at NNNNNr/onevision-phase2-code (private).
Contents
caches/
fused_tokens_25k/ # 48 GB · 25,188 × .pt (a8 mu tokens)
ff_feats_dinov2/ # 13 GB · 25,387 × .pt (DINOv2-L first-frame feats)… See the full description on the dataset page: https://huggingface.co/datasets/NNNNNr/onevision-phase2-data.phase1_sort_by_color_A1_10fps_via4cm
Phase1 Sort By Color A1 10Fps Via4Cm
LeRobot v3.0 dataset collected via
SCRAPE-IsaacLab — a
Code-as-Policies replay pipeline running inside Isaac Sim 5.1 / IsaacLab 2.3.2.
Task instruction: "Sort the blocks onto the matching colored dishes."
Robot: so101_follower
Cameras: top + left-wrist RGB @ 10 fps
Episodes: 100 (74,505 frames total)
Labels: per-frame natural-language skill labels in skill.natural_language
and subtask.* columns (labeled by Gemini)
Generated on Isaac Sim 5.1… See the full description on the dataset page: https://huggingface.co/datasets/HyeonseokE/phase1_sort_by_color_A1_10fps_via4cm.rq2_turn_off_lever_phase1_20_10fps
Rq2 Turn Off Lever Phase1 20 10Fps
LeRobot v3.0 dataset collected via
SCRAPE-IsaacLab — a
Code-as-Policies replay pipeline running inside Isaac Sim 5.1 / IsaacLab 2.3.2.
Task instruction: "Turn the lever off; the status indicator should turn red."
Robot: so101_follower
Cameras: top + left-wrist RGB @ 10 fps
Episodes: 20 (4,344 frames total)
Labels: per-frame natural-language skill labels in skill.natural_language
and subtask.* columns (labeled by Gemini)
Generated on Isaac Sim… See the full description on the dataset page: https://huggingface.co/datasets/HyeonseokE/rq2_turn_off_lever_phase1_20_10fps.IsaacLab-SO101-Phase1-turn_off_lever-80episode
Isaaclab So101 Phase1 Turn Off Lever 80Episode
LeRobot v3.0 dataset collected via
SCRAPE-IsaacLab — a
Code-as-Policies replay pipeline running inside Isaac Sim 5.1 / IsaacLab 2.3.2.
Task instruction: "Turn the lever off; the status indicator should turn red."
Robot: so101_follower
Cameras: top + left-wrist RGB @ 10 fps
Episodes: 80 (26,857 frames total)
Labels: per-frame natural-language skill labels in skill.natural_language
and subtask.* columns (labeled by Gemini)
Generated… See the full description on the dataset page: https://huggingface.co/datasets/Cache-SCA/IsaacLab-SO101-Phase1-turn_off_lever-80episode.store_play_v2_en_ablation_phase3phase1_push_button_A2_10fps
Phase1 Push Button A2 10Fps
LeRobot v3.0 dataset collected via
SCRAPE-IsaacLab — a
Code-as-Policies replay pipeline running inside Isaac Sim 5.1 / IsaacLab 2.3.2.
Task instruction: "Press the red button."
Robot: so101_follower
Cameras: top + left-wrist RGB @ 10 fps
Episodes: 100 (11,380 frames total)
Labels: per-frame natural-language skill labels in skill.natural_language
and subtask.* columns (labeled by Gemini)
Generated on Isaac Sim 5.1 / IsaacLab 2.3.2 under Python 3.11… See the full description on the dataset page: https://huggingface.co/datasets/HyeonseokE/phase1_push_button_A2_10fps.phase2_config_test_incomplete_v0.1_20260911_092705This dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v3.0",
"fps": 30,
"features": {
"action": {
"dtype": "float32",
"names": [
"left_shoulder_pan.pos",
"left_shoulder_lift.pos",
"left_elbow_flex.pos",
"left_wrist_flex.pos",
"left_wrist_roll.pos",
"left_gripper.pos"… See the full description on the dataset page: https://huggingface.co/datasets/OrderDraconis/phase2_config_test_incomplete_v0.1_20260911_092705.CLG_Laundry_TASK1_phase2phase1_pick_place_A1_10fps
Phase1 Pick Place A1 10Fps
LeRobot v3.0 dataset collected via
SCRAPE-IsaacLab — a
Code-as-Policies replay pipeline running inside Isaac Sim 5.1 / IsaacLab 2.3.2.
Task instruction: "Pick up the red block and place it on the blue dish."
Robot: so101_follower
Cameras: top + left-wrist RGB @ 10 fps
Episodes: 100 (28,459 frames total)
Labels: per-frame natural-language skill labels in skill.natural_language
and subtask.* columns (labeled by Gemini)
Generated on Isaac Sim 5.1 /… See the full description on the dataset page: https://huggingface.co/datasets/HyeonseokE/phase1_pick_place_A1_10fps.sort_by_color_ours_phase1_80_10fps
Sort By Color Ours Phase1 80 10Fps
LeRobot v3.0 dataset collected via
SCRAPE-IsaacLab — a
Code-as-Policies replay pipeline running inside Isaac Sim 5.1 / IsaacLab 2.3.2.
Task instruction: "Sort the blocks onto the matching colored dishes."
Robot: so101_follower
Cameras: top + left-wrist RGB @ 10 fps
Episodes: 80 (59,964 frames total)
Labels: per-frame natural-language skill labels in skill.natural_language
and subtask.* columns (labeled by Gemini)
Generated on Isaac Sim 5.1 /… See the full description on the dataset page: https://huggingface.co/datasets/HyeonseokE/sort_by_color_ours_phase1_80_10fps.phase1_pick_place_A2_10fps
Phase1 Pick Place A2 10Fps
LeRobot v3.0 dataset collected via
SCRAPE-IsaacLab — a
Code-as-Policies replay pipeline running inside Isaac Sim 5.1 / IsaacLab 2.3.2.
Task instruction: "Pick up the red block and place it on the blue dish."
Robot: so101_follower
Cameras: top + left-wrist RGB @ 10 fps
Episodes: 100 (31,526 frames total)
Labels: per-frame natural-language skill labels in skill.natural_language
and subtask.* columns (labeled by Gemini)
Generated on Isaac Sim 5.1 /… See the full description on the dataset page: https://huggingface.co/datasets/HyeonseokE/phase1_pick_place_A2_10fps.Challenge-phase2-rollouts-dataset
Challenge phase2 rollouts dataset
Policy rollout episodes collected for the RSS2026 Challenge (phase 2), in LeRobot v2.1 format.
Sub-datasets: 23 (one folder each)
Total episodes: 808 | Total frames: 4,289,763
Format: LeRobot codebase_version v2.1
Tasks: insert-mouse-battery, seal-water-bottle-cap, tower-of-hanoi-game
Policies: generalist / specialist (per team)
Each frame carries observation.commander_state distinguishing human teleoperation from autopilot policy inference… See the full description on the dataset page: https://huggingface.co/datasets/Posttraining-RFM-RSS2026/Challenge-phase2-rollouts-dataset.phase1_push_button_A1_10fps
Phase1 Push Button A1 10Fps
LeRobot v3.0 dataset collected via
SCRAPE-IsaacLab — a
Code-as-Policies replay pipeline running inside Isaac Sim 5.1 / IsaacLab 2.3.2.
Task instruction: "Press the red button."
Robot: so101_follower
Cameras: top + left-wrist RGB @ 10 fps
Episodes: 100 (11,320 frames total)
Labels: per-frame natural-language skill labels in skill.natural_language
and subtask.* columns (labeled by Gemini)
Generated on Isaac Sim 5.1 / IsaacLab 2.3.2 under Python 3.11… See the full description on the dataset page: https://huggingface.co/datasets/HyeonseokE/phase1_push_button_A1_10fps.phase1-subtask-green-0209This dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v3.0",
"robot_type": "so101_follower",
"total_episodes": 35,
"total_frames": 25497,
"total_tasks": 1,
"chunks_size": 1000,
"data_files_size_in_mb": 100,
"video_files_size_in_mb": 500,
"fps": 30,
"splits": {
"train": "0:35"
},
"data_path": "data/chunk-{chunk_index:03d}/file-{file_index:03d}.parquet",
"video_path":… See the full description on the dataset page: https://huggingface.co/datasets/Ryan-ZL-LIN/phase1-subtask-green-0209.pyramid-dataset-phase3_20260827_123302This dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v3.0",
"fps": 30,
"features": {
"action": {
"dtype": "float32",
"names": [
"shoulder_pan.pos",
"shoulder_lift.pos",
"elbow_flex.pos",
"wrist_flex.pos",
"wrist_roll.pos",
"gripper.pos"
],
"shape": [
6… See the full description on the dataset page: https://huggingface.co/datasets/heba000/pyramid-dataset-phase3_20260827_123302.pyramid-dataset-phase3_20260827_132836This dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v3.0",
"fps": 30,
"features": {
"action": {
"dtype": "float32",
"names": [
"shoulder_pan.pos",
"shoulder_lift.pos",
"elbow_flex.pos",
"wrist_flex.pos",
"wrist_roll.pos",
"gripper.pos"
],
"shape": [
6… See the full description on the dataset page: https://huggingface.co/datasets/heba000/pyramid-dataset-phase3_20260827_132836.