Arthur12137/SoftVTBench

1---2license: apache-2.03task_categories:4  - robotics5tags:6  - manipulation7  - tactile8  - visuo-tactile9  - deformable-objects10  - soft-body11  - libero12  - isaac-lab13pretty_name: SoftVTBench14size_categories:15  - 1K<n<10K16---17 18# SoftVTBench19 20Visuo-tactile manipulation demonstrations for rigid **and** soft/deformable21LIBERO-style pick-and-place tasks, collected with a tactile-sensing Franka arm22in Isaac Lab (Tabero simulation stack).23 24Mirrored on both hubs:25 26- Hugging Face — [`Arthur12137/SoftVTBench`](https://huggingface.co/datasets/Arthur12137/SoftVTBench)27- ModelScope — [`Arthur12137/SoftVTBench`](https://www.modelscope.cn/datasets/Arthur12137/SoftVTBench)28 29Earlier releases (evaluation USD assets and the first partial data drops) were30moved to `Arthur12137/SoftVTBench-archive` on both hubs.31 32## Contents33 34Four subsets, each **10 tasks × 100 successful demonstrations = 1000 demos**35and **4000 videos**.36 37| Subset | Scene / goal | Object | Video res. | Episode len | hdf5 | Videos | Total |38|---|---|---|---|---|---|---|---|39| `spatial-rigid` | kitchen table → plate | rigid pastry | 512×512 | 141–181 | 1250 MB | 560 MB | 1.8 GB |40| `object-rigid` | floor → basket | rigid pastry | 512×512 | 107–130 | 475 MB | 482 MB | 975 MB |41| `object-soft` | floor → basket | FEM soft body | 1024×1024 | 112–137 | 919 MB | 1074 MB | 2.0 GB |42| `spatial-soft` | kitchen table → plate | FEM soft body | 1024×1024 | 120–144 | 1390 MB | 1138 MB | 2.5 GB |43 44Every episode carries `success = True`; failures were filtered out at assembly45time (the skipped records are kept in `failure.jsonl`).46 47## Layout48 49```50<subset>/51├── manifest.jsonl              # one line per demo: task_id, demo_id, language, num_samples,52│                               # success, paths to the hdf5 and the 4 videos53├── assemble_summary.json       # per-task counts produced by the assembler54├── duplicate.jsonl             # empty in all four subsets55├── failure.jsonl               # demos dropped during assembly (soft subsets only)56└── libero_{spatial,object}/57    └── libero_{spatial,object}_task{0..9}/58        ├── replayed_demos/<task>_<language>_replayed_demo.hdf5     # 100 demos59        └── video_datasets/<task>/60            ├── videos/demo_<i>_agentview_rgb.mp461            ├── videos/demo_<i>_eye_in_hand_rgb.mp462            └── tactile_outputs/demo_<i>_gsmini_{left,right}_markers_rgb.mp463```64 65## HDF5 schema66 67```68data/demo_<i>/69├── actions            (T, 13) float32     # full action record70├── actions_binary     (T, 7)  float32     # 6-DoF delta + binary gripper71├── obs/                                   # policy-facing observations72├── soft_extras/                           # raw / extra channels used for analysis73├── states/                                # per-step full simulator state74└── initial_state/                         # reset state (unique per demo)75```76 77`obs/` fields, all four subsets:78 79`actions`, `applied_torque`, `arm_joint_pos`, `computed_torque`,80`eef_axis_angle`, `eef_pose`, `fem_bbox_dims`, `finger_force`,81`gripper_binary`, `gripper_close_norm`, `gripper_marker_motion`,82`gripper_net_force`, `gripper_pos`, `gripper_width`83 84The two soft subsets add four deformation channels:85`fem_deformation_max`, `fem_deformation_rms`,86`fem_kabsch_max_pct`, `fem_kabsch_rms_pct`.87 88`gripper_marker_motion` is the tactile signal: `(T, 2, 2, 99, 2)` — two GelSight89Mini pads × (reference, current) × 99 markers × (x, y) displacement.90 91Per-demo attributes include `asset_name`, `demo_id`, `language`, `num_samples`,92`success`, `soft_task_kind`, `task_id`, `task_suite`, and a `metadata_json`93blob with the sampled gripper closure.94 95## Verified integrity96 97Checked over the full release, not sampled unless noted:98 99- demo count = manifest lines = `success=True` count = 1000 per subset100- 4 videos per demo, no zero-byte or truncated files101- every path referenced by `manifest.jsonl` exists102- video frame count equals hdf5 episode length (360 pairs sampled)103- no NaN/Inf in any float field; all `obs/` fields aligned with `actions`104- 1000/1000 unique action trajectories and 1000/1000 unique `initial_state`105  per subset — the per-demo randomisation (object XY, goal XY, robot joints,106  gripper closing force) is genuinely distinct107 108## Known caveats109 110- **Video resolution differs**: rigid subsets are 512×512, soft subsets 1024×1024.111- **`demo_id` conventions differ** across subsets (9-digit global ids in112  `spatial-rigid`/`object-rigid`, 5–6-digit in `object-soft`, 0–350 in113  `spatial-soft`). In `spatial-soft` the id is only unique *within* a task.114  Index by `(subset, task_id, demo)` when merging subsets.115- **HDF5 compression is not uniform**: the two `object-*` subsets were written116  with gzip, the two `spatial-*` subsets uncompressed. This is why117  `object-rigid` is much smaller on disk despite holding the same amount of data.118- **`gripper_marker_motion` is stored twice**, identically, under both `obs/`119  and `soft_extras/`; it accounts for ~40% of each file.120- **Task 6 language is a placeholder** (`"golden pastry"`) in `object-rigid`121  and `object-soft`.122- Paths inside `manifest.jsonl` are relative to the original collection root123  (`outputs/...`), not to this repository.124 125## Download126 127```bash128pip install -U huggingface_hub129hf download Arthur12137/SoftVTBench --repo-type dataset --local-dir ./SoftVTBench130```131 132```bash133pip install -U modelscope134modelscope download --dataset Arthur12137/SoftVTBench --local_dir ./SoftVTBench135```136 137## License138 139Apache-2.0.140