datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
Challenge-phase1-dataset
Post-training for Robotics Foundation Models — Challenge Phase 1 Dataset
This is the public Phase 1 dataset for the RSS 2026 Workshop & Challenge on Post-training for Robotics Foundation Models.
Workshop website: https://posttraining-for-robotics.github.io/
The dataset contains real-robot bimanual manipulation trajectories on three benchmark tasks, collected on a bimanual YAM follower teleoperated by a GELLO leader arm. Every frame is timestamp-aligned across joint state, action… See the full description on the dataset page: https://huggingface.co/datasets/Posttraining-RFM-RSS2026/Challenge-phase1-dataset.phase1_sort_by_color_A1_10fps
Phase1 Sort By Color A1 10Fps
LeRobot v3.0 dataset collected via
SCRAPE-IsaacLab — a
Code-as-Policies replay pipeline running inside Isaac Sim 5.1 / IsaacLab 2.3.2.
Task instruction: "Sort the blocks onto the matching colored dishes."
Robot: so101_follower
Cameras: top + left-wrist RGB @ 10 fps
Episodes: 100 (74,322 frames total)
Labels: per-frame natural-language skill labels in skill.natural_language
and subtask.* columns (labeled by Gemini)
Generated on Isaac Sim 5.1 /… See the full description on the dataset page: https://huggingface.co/datasets/HyeonseokE/phase1_sort_by_color_A1_10fps.phase1_sort_by_color_A2_10fps
Phase1 Sort By Color A2 10Fps
LeRobot v3.0 dataset collected via
SCRAPE-IsaacLab — a
Code-as-Policies replay pipeline running inside Isaac Sim 5.1 / IsaacLab 2.3.2.
Task instruction: "Sort the blocks onto the matching colored dishes."
Robot: so101_follower
Cameras: top + left-wrist RGB @ 10 fps
Episodes: 100 (74,921 frames total)
Labels: per-frame natural-language skill labels in skill.natural_language
and subtask.* columns (labeled by Gemini)
Generated on Isaac Sim 5.1 /… See the full description on the dataset page: https://huggingface.co/datasets/HyeonseokE/phase1_sort_by_color_A2_10fps.turn_on_lever_ours_phase1_80_10fps
Turn On Lever Ours Phase1 80 10Fps
LeRobot v3.0 dataset collected via
SCRAPE-IsaacLab — a
Code-as-Policies replay pipeline running inside Isaac Sim 5.1 / IsaacLab 2.3.2.
Task instruction: "Turn the lever on; the status indicator should turn green."
Robot: so101_follower
Cameras: top + left-wrist RGB @ 10 fps
Episodes: 80 (17,457 frames total)
Labels: per-frame natural-language skill labels in skill.natural_language
and subtask.* columns (labeled by Gemini)
Generated on Isaac… See the full description on the dataset page: https://huggingface.co/datasets/HyeonseokE/turn_on_lever_ours_phase1_80_10fps.phase1_sort_by_color_A2_10fps_via4cm
Phase1 Sort By Color A2 10Fps Via4Cm
LeRobot v3.0 dataset collected via
SCRAPE-IsaacLab — a
Code-as-Policies replay pipeline running inside Isaac Sim 5.1 / IsaacLab 2.3.2.
Task instruction: "Sort the blocks onto the matching colored dishes."
Robot: so101_follower
Cameras: top + left-wrist RGB @ 10 fps
Episodes: 100 (74,827 frames total)
Labels: per-frame natural-language skill labels in skill.natural_language
and subtask.* columns (labeled by Gemini)
Generated on Isaac Sim 5.1… See the full description on the dataset page: https://huggingface.co/datasets/HyeonseokE/phase1_sort_by_color_A2_10fps_via4cm.close_box_ours_phase1_80_10fps
Close Box Ours Phase1 80 10Fps
LeRobot v3.0 dataset collected via
SCRAPE-IsaacLab — a
Code-as-Policies replay pipeline running inside Isaac Sim 5.1 / IsaacLab 2.3.2.
Task instruction: "Close the box by placing the lid on the box body."
Robot: so101_follower
Cameras: top + left-wrist RGB @ 10 fps
Episodes: 80 (22,566 frames total)
Labels: per-frame natural-language skill labels in skill.natural_language
and subtask.* columns (labeled by Gemini)
Generated on Isaac Sim 5.1 /… See the full description on the dataset page: https://huggingface.co/datasets/HyeonseokE/close_box_ours_phase1_80_10fps.open_box_ours_phase1_80_10fps
Open Box Ours Phase1 80 10Fps
LeRobot v3.0 dataset collected via
SCRAPE-IsaacLab — a
Code-as-Policies replay pipeline running inside Isaac Sim 5.1 / IsaacLab 2.3.2.
Task instruction: "Open the box by moving the lid to the target marker."
Robot: so101_follower
Cameras: top + left-wrist RGB @ 10 fps
Episodes: 80 (23,214 frames total)
Labels: per-frame natural-language skill labels in skill.natural_language
and subtask.* columns (labeled by Gemini)
Generated on Isaac Sim 5.1 /… See the full description on the dataset page: https://huggingface.co/datasets/HyeonseokE/open_box_ours_phase1_80_10fps.rq2_turn_off_lever_phase1_20_10fps
Rq2 Turn Off Lever Phase1 20 10Fps
LeRobot v3.0 dataset collected via
SCRAPE-IsaacLab — a
Code-as-Policies replay pipeline running inside Isaac Sim 5.1 / IsaacLab 2.3.2.
Task instruction: "Turn the lever off; the status indicator should turn red."
Robot: so101_follower
Cameras: top + left-wrist RGB @ 10 fps
Episodes: 20 (4,344 frames total)
Labels: per-frame natural-language skill labels in skill.natural_language
and subtask.* columns (labeled by Gemini)
Generated on Isaac Sim… See the full description on the dataset page: https://huggingface.co/datasets/HyeonseokE/rq2_turn_off_lever_phase1_20_10fps.xlerobot-desk-marker-phase1
saivishwak/xlerobot-desk-marker-phase1
This dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v3.0",
"fps": 30,
"features": {
"action": {
"dtype": "float32",
"shape": [
12
],
"names": [
"left_arm_shoulder_pan.pos",
"left_arm_shoulder_lift.pos",
"left_arm_elbow_flex.pos",
"left_arm_wrist_flex.pos",
"left_arm_wrist_roll.pos"… See the full description on the dataset page: https://huggingface.co/datasets/saivishwak/xlerobot-desk-marker-phase1.turn_off_lever_ours_phase1_80_10fps
Turn Off Lever Ours Phase1 80 10Fps
LeRobot v3.0 dataset collected via
SCRAPE-IsaacLab — a
Code-as-Policies replay pipeline running inside Isaac Sim 5.1 / IsaacLab 2.3.2.
Task instruction: "Turn the lever off; the status indicator should turn red."
Robot: so101_follower
Cameras: top + left-wrist RGB @ 10 fps
Episodes: 80 (17,232 frames total)
Labels: per-frame natural-language skill labels in skill.natural_language
and subtask.* columns (labeled by Gemini)
Generated on Isaac… See the full description on the dataset page: https://huggingface.co/datasets/HyeonseokE/turn_off_lever_ours_phase1_80_10fps.phase1_sort_by_color_A1_10fps_via4cm
Phase1 Sort By Color A1 10Fps Via4Cm
LeRobot v3.0 dataset collected via
SCRAPE-IsaacLab — a
Code-as-Policies replay pipeline running inside Isaac Sim 5.1 / IsaacLab 2.3.2.
Task instruction: "Sort the blocks onto the matching colored dishes."
Robot: so101_follower
Cameras: top + left-wrist RGB @ 10 fps
Episodes: 100 (74,505 frames total)
Labels: per-frame natural-language skill labels in skill.natural_language
and subtask.* columns (labeled by Gemini)
Generated on Isaac Sim 5.1… See the full description on the dataset page: https://huggingface.co/datasets/HyeonseokE/phase1_sort_by_color_A1_10fps_via4cm.IsaacLab-SO101-Phase1-turn_off_lever-80episode
Isaaclab So101 Phase1 Turn Off Lever 80Episode
LeRobot v3.0 dataset collected via
SCRAPE-IsaacLab — a
Code-as-Policies replay pipeline running inside Isaac Sim 5.1 / IsaacLab 2.3.2.
Task instruction: "Turn the lever off; the status indicator should turn red."
Robot: so101_follower
Cameras: top + left-wrist RGB @ 10 fps
Episodes: 80 (26,857 frames total)
Labels: per-frame natural-language skill labels in skill.natural_language
and subtask.* columns (labeled by Gemini)
Generated… See the full description on the dataset page: https://huggingface.co/datasets/Cache-SCA/IsaacLab-SO101-Phase1-turn_off_lever-80episode.sort_by_color_ours_phase1_80_10fps
Sort By Color Ours Phase1 80 10Fps
LeRobot v3.0 dataset collected via
SCRAPE-IsaacLab — a
Code-as-Policies replay pipeline running inside Isaac Sim 5.1 / IsaacLab 2.3.2.
Task instruction: "Sort the blocks onto the matching colored dishes."
Robot: so101_follower
Cameras: top + left-wrist RGB @ 10 fps
Episodes: 80 (59,964 frames total)
Labels: per-frame natural-language skill labels in skill.natural_language
and subtask.* columns (labeled by Gemini)
Generated on Isaac Sim 5.1 /… See the full description on the dataset page: https://huggingface.co/datasets/HyeonseokE/sort_by_color_ours_phase1_80_10fps.phase1_push_button_A2_10fps
Phase1 Push Button A2 10Fps
LeRobot v3.0 dataset collected via
SCRAPE-IsaacLab — a
Code-as-Policies replay pipeline running inside Isaac Sim 5.1 / IsaacLab 2.3.2.
Task instruction: "Press the red button."
Robot: so101_follower
Cameras: top + left-wrist RGB @ 10 fps
Episodes: 100 (11,380 frames total)
Labels: per-frame natural-language skill labels in skill.natural_language
and subtask.* columns (labeled by Gemini)
Generated on Isaac Sim 5.1 / IsaacLab 2.3.2 under Python 3.11… See the full description on the dataset page: https://huggingface.co/datasets/HyeonseokE/phase1_push_button_A2_10fps.phase1_pick_place_A1_10fps
Phase1 Pick Place A1 10Fps
LeRobot v3.0 dataset collected via
SCRAPE-IsaacLab — a
Code-as-Policies replay pipeline running inside Isaac Sim 5.1 / IsaacLab 2.3.2.
Task instruction: "Pick up the red block and place it on the blue dish."
Robot: so101_follower
Cameras: top + left-wrist RGB @ 10 fps
Episodes: 100 (28,459 frames total)
Labels: per-frame natural-language skill labels in skill.natural_language
and subtask.* columns (labeled by Gemini)
Generated on Isaac Sim 5.1 /… See the full description on the dataset page: https://huggingface.co/datasets/HyeonseokE/phase1_pick_place_A1_10fps.phase1-subtask-green-0209This dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v3.0",
"robot_type": "so101_follower",
"total_episodes": 35,
"total_frames": 25497,
"total_tasks": 1,
"chunks_size": 1000,
"data_files_size_in_mb": 100,
"video_files_size_in_mb": 500,
"fps": 30,
"splits": {
"train": "0:35"
},
"data_path": "data/chunk-{chunk_index:03d}/file-{file_index:03d}.parquet",
"video_path":… See the full description on the dataset page: https://huggingface.co/datasets/Ryan-ZL-LIN/phase1-subtask-green-0209.phase1_pick_place_A2_10fps
Phase1 Pick Place A2 10Fps
LeRobot v3.0 dataset collected via
SCRAPE-IsaacLab — a
Code-as-Policies replay pipeline running inside Isaac Sim 5.1 / IsaacLab 2.3.2.
Task instruction: "Pick up the red block and place it on the blue dish."
Robot: so101_follower
Cameras: top + left-wrist RGB @ 10 fps
Episodes: 100 (31,526 frames total)
Labels: per-frame natural-language skill labels in skill.natural_language
and subtask.* columns (labeled by Gemini)
Generated on Isaac Sim 5.1 /… See the full description on the dataset page: https://huggingface.co/datasets/HyeonseokE/phase1_pick_place_A2_10fps.phase1_push_button_A1_10fps
Phase1 Push Button A1 10Fps
LeRobot v3.0 dataset collected via
SCRAPE-IsaacLab — a
Code-as-Policies replay pipeline running inside Isaac Sim 5.1 / IsaacLab 2.3.2.
Task instruction: "Press the red button."
Robot: so101_follower
Cameras: top + left-wrist RGB @ 10 fps
Episodes: 100 (11,320 frames total)
Labels: per-frame natural-language skill labels in skill.natural_language
and subtask.* columns (labeled by Gemini)
Generated on Isaac Sim 5.1 / IsaacLab 2.3.2 under Python 3.11… See the full description on the dataset page: https://huggingface.co/datasets/HyeonseokE/phase1_push_button_A1_10fps.so101-paper-drag-phase1stack_2_cubes_ours_phase1_80_10fps
Stack 2 Cubes Ours Phase1 80 10Fps
LeRobot v3.0 dataset collected via
SCRAPE-IsaacLab — a
Code-as-Policies replay pipeline running inside Isaac Sim 5.1 / IsaacLab 2.3.2.
Task instruction: "Stack the green block on the red block."
Robot: so101_follower
Cameras: top + left-wrist RGB @ 10 fps
Episodes: 80 (30,403 frames total)
Labels: per-frame natural-language skill labels in skill.natural_language
and subtask.* columns (labeled by Gemini)
Generated on Isaac Sim 5.1 / IsaacLab… See the full description on the dataset page: https://huggingface.co/datasets/HyeonseokE/stack_2_cubes_ours_phase1_80_10fps.phase1_pick_place_A1_10fps_via4cm
Phase1 Pick Place A1 10Fps Via4Cm
LeRobot v3.0 dataset collected via
SCRAPE-IsaacLab — a
Code-as-Policies replay pipeline running inside Isaac Sim 5.1 / IsaacLab 2.3.2.
Task instruction: "Pick up the red block and place it on the blue dish."
Robot: so101_follower
Cameras: top + left-wrist RGB @ 10 fps
Episodes: 100 (28,530 frames total)
Labels: per-frame natural-language skill labels in skill.natural_language
and subtask.* columns (labeled by Gemini)
Generated on Isaac Sim 5.1… See the full description on the dataset page: https://huggingface.co/datasets/HyeonseokE/phase1_pick_place_A1_10fps_via4cm.pyramid-dataset-phase1_20260820_120343This dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v3.0",
"fps": 30,
"features": {
"action": {
"dtype": "float32",
"names": [
"shoulder_pan.pos",
"shoulder_lift.pos",
"elbow_flex.pos",
"wrist_flex.pos",
"wrist_roll.pos",
"gripper.pos"
],
"shape": [
6… See the full description on the dataset page: https://huggingface.co/datasets/heba000/pyramid-dataset-phase1_20260820_120343.phase1_pick_place_A2_10fps_via4cm
Phase1 Pick Place A2 10Fps Via4Cm
LeRobot v3.0 dataset collected via
SCRAPE-IsaacLab — a
Code-as-Policies replay pipeline running inside Isaac Sim 5.1 / IsaacLab 2.3.2.
Task instruction: "Pick up the red block and place it on the blue dish."
Robot: so101_follower
Cameras: top + left-wrist RGB @ 10 fps
Episodes: 100 (28,755 frames total)
Labels: per-frame natural-language skill labels in skill.natural_language
and subtask.* columns (labeled by Gemini)
Generated on Isaac Sim 5.1… See the full description on the dataset page: https://huggingface.co/datasets/HyeonseokE/phase1_pick_place_A2_10fps_via4cm.push_cube_ours_phase1_80_10fps
Push Cube Ours Phase1 80 10Fps
LeRobot v3.0 dataset collected via
SCRAPE-IsaacLab — a
Code-as-Policies replay pipeline running inside Isaac Sim 5.1 / IsaacLab 2.3.2.
Task instruction: "Push the cube to the target marker."
Robot: so101_follower
Cameras: top + left-wrist RGB @ 10 fps
Episodes: 80 (17,316 frames total)
Labels: per-frame natural-language skill labels in skill.natural_language
and subtask.* columns (labeled by Gemini)
Generated on Isaac Sim 5.1 / IsaacLab 2.3.2… See the full description on the dataset page: https://huggingface.co/datasets/HyeonseokE/push_cube_ours_phase1_80_10fps.redundancy_pickandplace_ours_phase1_100_10fps
Redundancy Pick Place Ours Phase1 100 10Fps
LeRobot v3.0 dataset collected via
SCRAPE-IsaacLab — a
Code-as-Policies replay pipeline running inside Isaac Sim 5.1 / IsaacLab 2.3.2.
Task instruction: "Pick up the red block and place it on the blue dish."
Robot: so101_follower
Cameras: top + left-wrist RGB @ 10 fps
Episodes: 100 (28,678 frames total)
Labels: per-frame natural-language skill labels in skill.natural_language
and subtask.* columns (labeled by Gemini)
Generated on… See the full description on the dataset page: https://huggingface.co/datasets/HyeonseokE/redundancy_pickandplace_ours_phase1_100_10fps.push_button_ours_phase1_80_10fps
Push Button Ours Phase1 80 10Fps
LeRobot v3.0 dataset collected via
SCRAPE-IsaacLab — a
Code-as-Policies replay pipeline running inside Isaac Sim 5.1 / IsaacLab 2.3.2.
Task instruction: "Press the red button."
Robot: so101_follower
Cameras: top + left-wrist RGB @ 10 fps
Episodes: 80 (9,110 frames total)
Labels: per-frame natural-language skill labels in skill.natural_language
and subtask.* columns (labeled by Gemini)
Generated on Isaac Sim 5.1 / IsaacLab 2.3.2 under Python… See the full description on the dataset page: https://huggingface.co/datasets/HyeonseokE/push_button_ours_phase1_80_10fps.redundancy_stack_2_cubes_ours_phase1_100_10fps
Redundancy Stack 2 Cubes Ours Phase1 100 10Fps
LeRobot v3.0 dataset collected via
SCRAPE-IsaacLab — a
Code-as-Policies replay pipeline running inside Isaac Sim 5.1 / IsaacLab 2.3.2.
Task instruction: "Stack the green block on the red block."
Robot: so101_follower
Cameras: top + left-wrist RGB @ 10 fps
Episodes: 100 (38,418 frames total)
Labels: per-frame natural-language skill labels in skill.natural_language
and subtask.* columns (labeled by Gemini)
Generated on Isaac Sim 5.1… See the full description on the dataset page: https://huggingface.co/datasets/HyeonseokE/redundancy_stack_2_cubes_ours_phase1_100_10fps.Phase_1This dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v2.1",
"robot_type": "so101_follower",
"total_episodes": 23,
"total_frames": 4837,
"total_tasks": 1,
"total_videos": 46,
"total_chunks": 1,
"chunks_size": 1000,
"fps": 30,
"splits": {
"train": "0:23"
},
"data_path": "data/chunk-{episode_chunk:03d}/episode_{episode_index:06d}.parquet",
"video_path":… See the full description on the dataset page: https://huggingface.co/datasets/Tristan-Lemelle/Phase_1.pull_cube_ours_phase1_80_10fps
Pull Cube Ours Phase1 80 10Fps
LeRobot v3.0 dataset collected via
SCRAPE-IsaacLab — a
Code-as-Policies replay pipeline running inside Isaac Sim 5.1 / IsaacLab 2.3.2.
Task instruction: "Pull the cube to the target marker."
Robot: so101_follower
Cameras: top + left-wrist RGB @ 10 fps
Episodes: 80 (25,568 frames total)
Labels: per-frame natural-language skill labels in skill.natural_language
and subtask.* columns (labeled by Gemini)
Generated on Isaac Sim 5.1 / IsaacLab 2.3.2… See the full description on the dataset page: https://huggingface.co/datasets/HyeonseokE/pull_cube_ours_phase1_80_10fps.extract_cube_ours_phase1_80_10fps
Extract Cube Ours Phase1 80 10Fps
LeRobot v3.0 dataset collected via
SCRAPE-IsaacLab — a
Code-as-Policies replay pipeline running inside Isaac Sim 5.1 / IsaacLab 2.3.2.
Task instruction: "Extract the cube from the pocket and place it on the target marker."
Robot: so101_follower
Cameras: top + left-wrist RGB @ 10 fps
Episodes: 80 (25,138 frames total)
Labels: per-frame natural-language skill labels in skill.natural_language
and subtask.* columns (labeled by Gemini)
Generated on… See the full description on the dataset page: https://huggingface.co/datasets/HyeonseokE/extract_cube_ours_phase1_80_10fps.
