wushr-lance/VLA2Vec
VLA2Vec Teleoperated bimanual manipulation data collected on a dual-arm + dexterous-hand robot (task: pick fruits). Directory structure pick_fruits/ episode_0000/ episode_0000.h5 # states, targets, timestamps (see below) episode_0000_head_left_rgb.mp4 # head camera, left eye episode_0000_left_wrist.mp4 # left wrist camera episode_0000_right_wrist.mp4 # right wrist camera episode_0001/ ... 100 episodes, all… See the full description on the dataset page: https://huggingface.co/datasets/wushr-lance/VLA2Vec.
VLA2Vec
Teleoperated bimanual manipulation data collected on a dual-arm + dexterous-hand robot (task: pick fruits).
Directory structure
pick_fruits/
episode_0000/
episode_0000.h5 # states, targets, timestamps (see below)
episode_0000_head_left_rgb.mp4 # head camera, left eye
episode_0000_left_wrist.mp4 # left wrist camera
episode_0000_right_wrist.mp4 # right wrist camera
episode_0001/
...100 episodes, all success trials. Tactile sensor videos (*_tactile_raw.mkv, *_tactile_deform.mkv) are not included in this upload.
Video frame k corresponds to hdf5 row k (same indexing, no separate frame-index field needed).
episode_XXXX.h5 fields
All arrays are indexed by timestep T (varies per episode, recorded at command_hz = 30).
Timestamps
Per-arm state & targets
Each field below exists twice, prefixed left_ and right_:
Episode-level attributes (h5py.File.attrs)
Action space (as consumed by policy training)
The raw h5 fields aren't used as the action directly. The standard representation is 62-D, chunked, delta-arm / absolute-hand:
side action (31D) = [delta_xyz(3), delta_rot6d(6), hand_target_joint_positions(22)]
bimanual action (62D) = concat(left[31], right[31]) # order: (left, right)
action chunk = stack of `action_horizon` steps # (action_horizon, 62); horizon=16 by convention- Hands (22D each):
{L/R}_hand_target_joint_positions[t], used as-is — absolute. - Arms (9D each: translation + rot6d): a delta pose, in the end-effector-local frame, relative to one reference shared by every step of the chunk — the measured end-effector pose at the chunk's first step (
{L/R}_arm_current_pose[i]), not re-measured per step and not chained onto the previous step's target.
def rot_to_6d(R): # first two columns (Zhou et al. 2019)
return np.concatenate([R[:, 0], R[:, 1]])
def delta_pose_9d(ref_pose, target_pose):
R_ref, t_ref = ref_pose[:3, :3], ref_pose[:3, 3]
R_tgt, t_tgt = target_pose[:3, :3], target_pose[:3, 3]
delta_xyz = R_ref.T @ (t_tgt - t_ref) # translation delta, EEF-local frame
R_delta = R_ref.T @ R_tgt # target_R = R_ref @ R_delta
return np.concatenate([delta_xyz, rot_to_6d(R_delta)])
def build_action_chunk(h5, side, i, horizon=16):
ref, T = h5[f"{side}_arm_current_pose"][i], h5.attrs["total_steps"]
chunk = []
for k in range(horizon):
t = min(i + k, T - 1)
d9 = delta_pose_9d(ref, h5[f"{side}_arm_target_pose"][t])
chunk.append(np.concatenate([d9, h5[f"{side}_hand_target_joint_positions"][t]]))
return np.stack(chunk) # (horizon, 31)
action_chunk = np.concatenate( # (horizon, 62)
[build_action_chunk(h5, "left", i), build_action_chunk(h5, "right", i)], axis=-1
)Loading example
import h5py
import cv2
ep = "pick_fruits/episode_0010/episode_0010"
f = h5py.File(f"{ep}.h5", "r")
left_target_pose = f["left_arm_target_pose"][:] # (T, 4, 4)
left_hand_target = f["left_hand_target_joint_positions"][:] # (T, 22)
T = f.attrs["total_steps"]
cap = cv2.VideoCapture(f"{ep}_head_left_rgb.mp4")
ok, frame_0 = cap.read() # frame k == h5 row k