jatshi/SafeVLA-Bench-Dataset

SafeVLA PickCube dataset card

Code, converters, split audit, and benchmark reports: <https://github.com/Jatshi/SafeVLA-Bench>.

Contents

• —pickcube_state_h16_full1000_holdout_v1.npz: 62,681 state/action-chunk samples from 1,000 CPU replays after excluding the five evaluation-seed episodes.
• —pickcube_state_h16_v1.npz: 6,220 state/action-chunk samples from the first 100 CPU replays; retained only for diagnostic comparison, not the final result.
• —pickcube_rgbd_h8_gpu_v1.npz: 7,902 RGB-D/state/action-chunk samples from 88 successful GPU replays.
• —lerobot_pickcube_state_v1/: LeRobot Dataset v3, 100 episodes, 7,720 frames, 20 FPS.
• —Scenario configs and result episode CSVs for SafeZone and ambiguity splits.

Source and processing

The source demonstrations are from the official ManiSkill PickCube motion-planning release. Conversion changes the controller to pd_ee_delta_pos, records observations, and forms fixed-horizon action chunks. No private user audio or personal data is included.

Known issue

ManiSkill GPU replay reports that reset options are ignored. The GPU RGB-D data therefore must not be interpreted as exact paired replays of the CPU initial states. The repository retains both logs and backend-specific hashes.

Splits

Evaluation uses seeds 11, 23, 37, 53, and 71. The final training conversion reads the replay JSON metadata and removes every episode whose episode_seed is in that set: five episodes are excluded from the 1,000-demo source before producing 62,681 H16 samples. SHA-256 for the strict NPZ is e18781cc75051d19ee0127b633fa340edf9443db403802bb73de5513a95ac664.

An audit found that the earlier first-100 training subset contained evaluation seeds. Its results are explicitly classified as a leakage-discovery pilot and cannot support final claims. Calibration fits on 11/23/37 and evaluates on 53/71. All corruptions of the same source episode must remain in the same split.

Safety and limitations

Danger labels refer to a simulated no-go sphere, not real-world injury risk. The dataset is suitable for reproducibility and education, not certification or physical deployment.

The SafeVLA-Bench source code is Apache-2.0. ManiSkill states that its rigid-body environments use permissive licenses while its visual assets are CC BY-NC 4.0. To avoid overstating downstream rights, the released derived trajectories and videos are marked CC BY-NC 4.0; users must also follow ManiSkill's third-party asset notices.