Arthur12137/SoftVTBench

SoftVTBench

Visuo-tactile manipulation demonstrations for rigid and soft/deformable LIBERO-style pick-and-place tasks, collected with a tactile-sensing Franka arm in Isaac Lab (Tabero simulation stack).

Mirrored on both hubs:

• —Hugging Face — `Arthur12137/SoftVTBench`
• —ModelScope — `Arthur12137/SoftVTBench`

Earlier releases (evaluation USD assets and the first partial data drops) were moved to Arthur12137/SoftVTBench-archive on both hubs.

Contents

Four subsets, each 10 tasks × 100 successful demonstrations = 1000 demos and 4000 videos.

Every episode carries success = True; failures were filtered out at assembly time (the skipped records are kept in failure.jsonl).

Layout

<subset>/
├── manifest.jsonl              # one line per demo: task_id, demo_id, language, num_samples,
│                               # success, paths to the hdf5 and the 4 videos
├── assemble_summary.json       # per-task counts produced by the assembler
├── duplicate.jsonl             # empty in all four subsets
├── failure.jsonl               # demos dropped during assembly (soft subsets only)
└── libero_{spatial,object}/
    └── libero_{spatial,object}_task{0..9}/
        ├── replayed_demos/<task>_<language>_replayed_demo.hdf5     # 100 demos
        └── video_datasets/<task>/
            ├── videos/demo_<i>_agentview_rgb.mp4
            ├── videos/demo_<i>_eye_in_hand_rgb.mp4
            └── tactile_outputs/demo_<i>_gsmini_{left,right}_markers_rgb.mp4

HDF5 schema

data/demo_<i>/
├── actions            (T, 13) float32     # full action record
├── actions_binary     (T, 7)  float32     # 6-DoF delta + binary gripper
├── obs/                                   # policy-facing observations
├── soft_extras/                           # raw / extra channels used for analysis
├── states/                                # per-step full simulator state
└── initial_state/                         # reset state (unique per demo)

obs/ fields, all four subsets:

actions, applied_torque, arm_joint_pos, computed_torque, eef_axis_angle, eef_pose, fem_bbox_dims, finger_force, gripper_binary, gripper_close_norm, gripper_marker_motion, gripper_net_force, gripper_pos, gripper_width

The two soft subsets add four deformation channels: fem_deformation_max, fem_deformation_rms, fem_kabsch_max_pct, fem_kabsch_rms_pct.

gripper_marker_motion is the tactile signal: (T, 2, 2, 99, 2) — two GelSight Mini pads × (reference, current) × 99 markers × (x, y) displacement.

Per-demo attributes include asset_name, demo_id, language, num_samples, success, soft_task_kind, task_id, task_suite, and a metadata_json blob with the sampled gripper closure.

Verified integrity

Checked over the full release, not sampled unless noted:

• —demo count = manifest lines = success=True count = 1000 per subset
• —4 videos per demo, no zero-byte or truncated files
• —every path referenced by manifest.jsonl exists
• —video frame count equals hdf5 episode length (360 pairs sampled)
• —no NaN/Inf in any float field; all obs/ fields aligned with actions
• —1000/1000 unique action trajectories and 1000/1000 unique initial_state per subset — the per-demo randomisation (object XY, goal XY, robot joints, gripper closing force) is genuinely distinct

Known caveats

• —Video resolution differs: rigid subsets are 512×512, soft subsets 1024×1024.
• —`demo_id` conventions differ across subsets (9-digit global ids in spatial-rigid/object-rigid, 5–6-digit in object-soft, 0–350 in spatial-soft). In spatial-soft the id is only unique within a task. Index by (subset, task_id, demo) when merging subsets.
• —HDF5 compression is not uniform: the two object-* subsets were written with gzip, the two spatial-* subsets uncompressed. This is why object-rigid is much smaller on disk despite holding the same amount of data.
• —`gripper_marker_motion` is stored twice, identically, under both obs/ and soft_extras/; it accounts for ~40% of each file.
• —Task 6 language is a placeholder ("golden pastry") in object-rigid and object-soft.
• —Paths inside manifest.jsonl are relative to the original collection root (outputs/...), not to this repository.

Download

pip install -U huggingface_hub
hf download Arthur12137/SoftVTBench --repo-type dataset --local-dir ./SoftVTBench

pip install -U modelscope
modelscope download --dataset Arthur12137/SoftVTBench --local_dir ./SoftVTBench

License

Apache-2.0.