Datasetpublic

SberRoboticsCenter/GreenChallengeData

GreenChallengeData Demonstrations of a bimanual humanoid robot performing manipulation tasks in simulation, for training vision-language-action (VLA) policies. The repository gathers four collections: three task-specific deliveries recorded by human teleoperation and augmented with synthetic trajectories, plus a large set of scripted-policy episodes covering 15 tasks. Format: LeRobot v2.1 · 30 fps · 3 cameras, 448×448 (H.264) · 51-dim state, 52-dim action Total: 24,055 episodes… See the full description on the dataset page: https://huggingface.co/datasets/SberRoboticsCenter/GreenChallengeData.

sourceHugging Facemitupdated 17d agoView on Hugging Face

1likes3.6kdownloads

Dataset Card

GreenChallengeData

Demonstrations of a bimanual humanoid robot performing manipulation tasks in simulation, for training vision-language-action (VLA) policies. The repository gathers four collections: three task-specific deliveries recorded by human teleoperation and augmented with synthetic trajectories, plus a large set of scripted-policy episodes covering 15 tasks.

Format: LeRobot v2.1 · 30 fps · 3 cameras, 448×448 (H.264) · 51-dim state, 52-dim action Total: 24,055 episodes · 15,169,390 frames (≈140 h) · 72,165 videos · ≈594 GB

Collection	Task	Episodes	Frames	Hours
`teleop_kitchen_tea_set`	cup and tea can onto a tray (kitchen)	4,367	2,134,571	19.8
`teleop_darkstore`	misplaced soda can back onto its shelf (warehouse)	5,957	1,138,544	10.5
`teleop_kitchen_plate`	plate onto a tray (kitchen)	4,587	1,150,811	10.7
`scripted`	15 tasks across 5 scenes	9,144	10,745,464	99.5
Total		24,055	15,169,390	140.5

Repository layout

teleop_kitchen_tea_set/       task-specific delivery
  teleop/<dataset>/           human teleoperation, one LeRobot dataset per recording session
  mimic/<dataset>/            synthetic trajectories in the same scene
teleop_darkstore/             same structure
teleop_kitchen_plate/         same structure
scripted/
  vla/                        LeRobot dataset (data/, videos/, meta/)
  metadata/refs/              sha256 manifest of every file

Every <dataset> directory is a self-contained LeRobot v2.1 dataset:

data/chunk-XXX/episode_NNNNNN.parquet          telemetry, chunks of 1000 episodes
videos/chunk-XXX/<camera>/episode_NNNNNN.mp4   observation.images.cam_head
                                               observation.images.cam_left_wrist
                                               observation.images.cam_right_wrist
meta/info.json                                 schema, joint_names, fps, totals
meta/episodes.jsonl                            per-episode: length, task_index, action_config, validity masks
meta/subtasks.jsonl                            per-frame subtask annotation
meta/tasks.jsonl                               task text
meta/episodes_stats.jsonl                      per-episode statistics for normalisation
meta/validation_episodes.json                  suggested validation split

Episode indices restart from 0 inside each <dataset>; there is no global numbering across the repository.

Collections

teleopkitchentea_set

Kitchen scene: pick a cup with the right hand and a tea can with the left, place both on a tray.

Dataset	Kind	Episodes	Frames
`teleop/kitchen_xr_lerobot_super_robot_20260831T182822Z-cbwIYi__success189`	teleop	189	86,218
`teleop/super-robot-v3-c-0Q9B-six-world-pose-20260829-wrist-rot180-v2`	teleop	38	20,332
`mimic/super-robot-mimic-v3-candidate36-fleet-1896-20260901-v1`	mimic	1,896	1,011,303
`mimic/super-robot-mimic-v3-candidate36-wide-30-20260830-wrist-rot180-v2`	mimic	30	15,990
`mimic/super-robot-mimic-candidate65-fleet-2214-20260904-v1`	mimic	2,214	1,000,728

Task text for the first four datasets: Pick up the cup with the right hand and the Tea can with the left hand, then place both items upright on the tray, with six subtasks:

Raise your right hand → Pick the cup from the shelf with your right hand → Place the cup upright onto the tray with your right hand → Raise your left hand → Pick the Tea can from the shelf with your left hand → Place the Tea can upright onto the tray with your left hand

candidate65 uses the same scene but a different scenario and annotation: Place the cup and the tea can on the tray, four subtasks without the Raise steps, and the tea can ends up lying on the tray rather than standing upright. Treat it as a separate task rather than extra episodes of the one above.

teleop_darkstore

Warehouse scene: move a misplaced red soda can from the snacks shelf to a free slot on the drinks shelf, right hand only.

Dataset	Kind	Episodes	Frames
`teleop/darkstore_v3_lerobot_all360__super-darkstore-two-world-pose-v1-merged`	teleop	360	72,552
`teleop/super-robot-darkstore-d3-TYBs-20260831`	teleop	56	9,460
`teleop/super-robot-darkstore-d3-wV2d-20260831`	teleop	29	5,935
`mimic/super-robot-darkstore-mimic-top3-5512-20260901-v1`	mimic	5,512	1,050,597

Task text: Move the misplaced red soda can to a free position on the Drinks shelf, two subtasks:

Pick the bottle from the Snacks shelf with your right hand → Place the bottle onto a Drinks shelf next to other bottles with your right hand

In this scene the head camera is tilted upwards, so the manipulated object is mostly visible in the right wrist camera.

teleopkitchenplate

Kitchen scene: take a plate from the dish drying rack and place it on a tray, left hand only.

Dataset	Kind	Episodes	Frames
`teleop/kitchen_e3_e-fNw__success401__super-kitchen-e3-left-plate-world-pose-v1`	teleop	401	86,895
`teleop/super-robot-kitchen-e3-teleop-e-F5A-super-20260903`	teleop	72	13,549
`mimic/super-robot-kitchen-e3-mimic-top5-4114-20260903-v1`	mimic	4,114	1,050,367

Task text: Raise your left hand, pick up the plate, place the plate onto the tray, three subtasks:

Raise your left hand → Pick the plate from the dish drying rack with your left hand → Place the plate onto the tray with your left hand

scripted

9,144 episodes generated by a scripted policy in a digital twin, covering 15 tasks in four scenes (kitchen, shop, warehouse and a room with two tables). Episodes are long — median 1,087 frames (36 s), maximum 2,932 (98 s) — and each is annotated with 2–6 subtasks.

Task	Scene	Episodes
Lift the lid of the Sber Ring box with your right hand, place the lid aside with your right hand, push the Sber Ring box towards the visitor with your right hand, move the Sber Ring box back with your right hand	shop	1,175
Open the lower drawer of the left cabinet with your right hand, take the cat toy with your left hand, close the drawer with your right hand, place the cat toy into the basket with your left hand	shop	1,141
Raise your right hand, pick up the cat toy with your right hand, place the cat toy inside the basket with your right hand, move your right hand away from the basket	shop	1,058
Raise your right hand, pick up the sugar bowl with your right hand, place the sugar bowl onto the tray with your right hand	kitchen	978
Raise your left hand, pick up the plate with your left hand	kitchen	886
Pick up the cup with your right hand, place the cup on the tray, pick up the tea can with your left hand, then place the tea can on the tray	kitchen	778
Pick the red soda can up with your right hand, place the red soda can in the empty space next to the matching red soda cans	warehouse	776
Approach the basket, Pick the basket from the table with your left hand, Place the basket onto another table with your left hand, Move your right hand out of the basket	warehouse	770
Pick up the cup with your right hand and place the cup onto the tray with your right hand	kitchen	506
Approach the basket, pick it up, and move it to another table	warehouse	457
Raise your left hand, pick up the plate, place the plate onto the tray	kitchen	264
Approach the basket, pick the basket from the table with your left hand, place the basket onto another table with your left hand, move your left hand out of the basket	room	172
Pick up the tea can with your left hand and place the tea can on the tray	warehouse	104
Raise your left hand, pick up the plate with your left hand, place the plate onto the tray with your left hand, move your left hand out of the plate	kitchen	42
Raise your left hand, pick up the plate with your left hand, place the plate onto the tray, pick up the sugar bowl with your right hand, place the sugar bowl onto the tray	kitchen	37

In the basket tasks the robot walks between two tables; in the other tasks it stays in place.

Data fields

All four collections share the same observation and action layout.

Field	Dim	Description
`observation.state`	51	0–11 legs · 12 torsoyawjoint · 13–17 left arm (shoulder pitch/roll/yaw, elbow pitch/yaw) · 18–22 right arm · 23–24 neck (yaw, pitch) · 25–26 left wrist (roll, pitch) · 27–28 right wrist · 29–34 left hand (pinky, ring, middle, index, thumb pitch, thumb yaw) · 35–40 right hand · 41–43 torso roll/pitch/z · 44 torso yaw · 45–50 base velocities
`action`	52	`action[:51]` is the target pose at the next frame, in the same layout as `state`; `action[51]` is episode progress 0→1
`observation.robot_command_35x.position`	35	controller command for the upper body (arm and neck targets); leads the state by a few frames
`timestamp`	1	`frame_index / 30`
`frame_index`, `episode_index`, `index`, `task_index`	1	indices

Joint names are listed in meta/info.json → joint_names. Hand values: an open palm is ≈ −0.1 summed over the six joints of a hand, a closed fist ≈ 6.3; a grasp lies in between depending on object size.

Validity masks (state_valid_mask / action_valid_mask in meta/episodes.jsonl) mark channels that carry no signal in a given episode, and they differ between collections:

—teleoperation datasets: legs (0–11) and channels 41–50 are invalid;
—mimic datasets: channels 41–50 are invalid;
—scripted: channels 45–50 are invalid, or 46–49 in the walking tasks, where linear X and angular Z velocity are meaningful. Legs are valid and do move.

task_index is a plain index within each teleoperation delivery, and a 64-bit hash of the task text in scripted. In scripted the index column is episode_index·10⁹ + frame_index rather than a running counter.

Subtask annotation

Every episode is segmented into subtasks with frame boundaries, in meta/subtasks.jsonl:

json

{"episode_index": 0,
 "subtasks": [{"descriptions": ["Raise your left hand"],
               "start_frame": 0, "end_frame": 319, "skill": "Raise"}, ...]}

Segments cover the episode end to end with no gaps or overlaps. The same segmentation is mirrored in the action_config field of meta/episodes.jsonl. For language-conditioned training, take instructions verbatim from descriptions; for the whole-task text, use meta/tasks.jsonl indexed by task_index.

Additional metadata

The teleoperation collections carry extra files produced by the recording pipeline:

—meta/world_poses.jsonl — per-frame 6-DoF poses of every scene object, of the tray/target guides and of 126 robot links, plus wall-clock timestamps. This is the only source for verifying task success; it is large (≈72 GB in total across the repository) and is not needed for behaviour cloning.
—meta/format_warnings.jsonl — soft warnings raised by the delivery validator, mainly short subtask segments.
—meta/subtask_projection_provenance.jsonl — how the subtask boundaries were derived.
—meta/super_format.json — delivery profile and task text.

scripted instead ships metadata/refs/current-data.json with the sha256 of every file, and its episodes_stats.jsonl contains telemetry statistics only — compute image statistics separately if you normalise the visual input.

Usage

python

from huggingface_hub import snapshot_download

# one dataset
path = snapshot_download(
    repo_id="SberRoboticsCenter/GreenChallengeData",
    repo_type="dataset",
    allow_patterns="teleop_kitchen_plate/teleop/kitchen_e3_e-fNw__success401__*/**",
)

# everything except the large world-pose files
path = snapshot_download(
    repo_id="SberRoboticsCenter/GreenChallengeData",
    repo_type="dataset",
    ignore_patterns="*world_poses.jsonl",
)

Each <dataset> directory is read by the standard LeRobot v2.1 loader. Video frame t corresponds to the parquet row with frame_index = t.