Tsaochengyu/datasetSEGMENTfordraw

DeliverStraw Drawer-Opening Segments

This dataset contains the drawer-opening portion extracted from all 504 target RoboCasa DeliverStraw demonstrations. It is derived from target/composite/DeliverStraw/20250813 in NVIDIA's PhysicalAI-Robotics-Manipulation-Kitchen-Demos release.

Extraction rule

For every source episode:

1. 1.Restore every recorded MuJoCo state with that episode's exact model.xml.gz and ep_meta.json.
2. 2.Find the first recorded contact between a Panda gripper finger collision geometry and the target drawer handle collision geometry.
3. 3.Start the segment five frames before that contact.
4. 4.Find the first subsequent frame where normalized drawer opening is at least 0.95.
5. 5.End the segment five frames after that opening frame.

The extraction uses recorded simulation states. It does not infer contact from images and does not reconstruct states by replaying actions.

Handle Frame

The Handle Frame is evaluated independently at every source frame:

• —origin: center of the moving drawer handle;
• —x axis: drawer width direction;
• —y axis: drawer depth direction;
• —z axis: drawer up direction.

The frame orientation follows the moving drawer body.

Default model interface

The standard proprioceptive input and action are both spatially aligned to the current Handle Frame. The three camera images remain in their native camera views.

The standard 16D observation.state keeps the original state layout while expressing both poses in the current Handle Frame:

[base_position_in_current_handle_frame(3),
 base_rotation_quaternion_xyzw_in_current_handle_frame(4),
 current_eef_position_in_current_handle_frame(3),
 current_eef_rotation_quaternion_xyzw_in_current_handle_frame(4),
 gripper_qpos(2)]

The standard 15D action retains the full action layout:

[base_motion_in_handle_axes(4), control_mode(1),
 commanded_eef_position_in_current_handle_frame(3),
 commanded_eef_x_axis_in_current_handle_frame(3),
 commanded_eef_y_axis_in_current_handle_frame(3),
 gripper(1)]

All Drawer Opening Segments use arm mode: the original four base-action values are zero in all 35,736 frames and the control-mode value is always -1. These dimensions are nevertheless retained in the standard action. A zero base-motion vector is invariant to the Handle Frame axis change; torso motion and control mode are scalar quantities without a spatial origin.

Additional input columns:

• —observation.handle_relative_state: explicit copy of the standard 16D Handle-Relative state.
• —observation.handle_relative_state_6d: 20D full state using position and 6D orientation for both the base and end effector, plus two gripper joints.
• —observation.handle_relative_eef_state: compact 8D end-effector-only input using a rotation vector.
• —observation.handle_relative_eef_state_6d: compact 11D end-effector-only input using 6D orientation.
• —observation.state.original: original 16D robot/world-frame proprioception.

Additional action columns:

• —action.handle_relative_target_6d: explicit copy of the standard 15D action.
• —action.handle_relative_eef_target_6d: compact 10D action without base and control-mode dimensions.
• —action.handle_relative_eef_target: 7D target using position, rotation vector, and gripper command.
• —action.handle_relative_target: preserved 12D full-layout target using a rotation vector.
• —action.original and action.robot_base_delta: the original 12D normalized controller delta in the robot base frame.
• —action.handle_component_relative_target: legacy 12D target using the old handle-component axes, retained for traceability.

observation.handle_relative_achieved_pose and its 6D variant contain the next recorded achieved end-effector pose. They are evaluation labels, not current-frame model inputs.

The 6D standard fields are recommended for behavior cloning because they avoid the rotation-vector branch discontinuity near 180 degrees.

Contents

• —504 valid episodes; 0 invalid episodes
• —35,736 frames at 20 FPS
• —504 per-episode parquet files
• —1,512 clipped videos from three cameras
• —504 clipped states.npz files
• —504 original per-episode model.xml.gz files
• —audit.jsonl with source/output frame mappings and physical event boundaries

Segment lengths range from 29 to 270 frames. For every episode, the contact is at output offset 5 and the final frame is five frames after the drawer first crosses the opening threshold.

Reproduction

The extraction implementation is in scripts/process_deliver_straw.py. The domain terminology and design decisions are recorded in docs/CONTEXT.md and docs/adr/.

python scripts/process_deliver_straw.py \
  --input DeliverStraw-20250813 \
  --output DeliverStraw-drawer-opening \
  --episodes all \
  --overwrite

python scripts/align_deliver_straw_object_frame.py \
  --dataset DeliverStraw-drawer-opening

Validation

All 504 outputs were checked for:

• —aligned parquet, simulation-state, and video frame counts;
• —continuous global and per-episode indices;
• —finite original actions, target actions, and achieved poses;
• —equality of observation.state and observation.handle_relative_state;
• —equality of action and action.handle_relative_target_6d;
• —equality of action.original and action.robot_base_delta;
• —unit-length, orthogonal x/y axes in every 6D rotation;
• —drawer-aligned current input and target output poses;
• —preserved zero base action and arm-only control mode throughout the data;
• —exactly three 20 FPS videos per episode;
• —exact contact-minus-start and end-minus-open offsets.

No consistency errors were found.

License and attribution

The source dataset is distributed under CC BY 4.0. Users should also follow the RoboCasa and robosuite software licenses when replaying simulation states.