qingzhu-robotics/TacRich-Manip-LeRobot-teleoperation-insert-peg-cylinder

TacRich-Manip LeRobot v3 — insertpegcylinder

Multimodal real-robot trajectories for gripper-based contact-rich manipulation, published in the standard LeRobot v3 layout. This task repository contains teleoperation/insert_peg_cylinder and is private during active collection.

Dataset summary

Robot teleoperation trajectories. action is the migrated commanded absolute flange target. observation.state_gripper and action_gripper are 10D gripper-tip poses from Tbasegripper=Tbaseflange@Tflangegripper with translation [0,0,0.2] m and flange-frame Rz(40 deg).

Task labels in meta/tasks.parquet:

• —teleoperation/insert_peg_cylinder

Complete feature inventory

Shapes below are the logical shapes recorded in meta/info.json. RGB video is stored as MP4 and decoded on demand; other frame fields are stored in Parquet.

The four primary vectors are ordered exactly as follows:

observation.state[15] =
  [arm_j1, arm_j2, arm_j3, arm_j4, arm_j5, arm_j6, arm_j7,
   flange_x_m, flange_y_m, flange_z_m,
   flange_qx, flange_qy, flange_qz, flange_qw,
   gripper_distance_mm]

action[8] =
  [target_flange_x_m, target_flange_y_m, target_flange_z_m,
   target_flange_qx, target_flange_qy, target_flange_qz, target_flange_qw,
   target_gripper_distance_mm]

observation.state_gripper[10] =
  [tip_x_m, tip_y_m, tip_z_m,
   R00, R10, R20, R01, R11, R21,
   gripper_distance_mm]

action_gripper[10] =
  [target_tip_x_m, target_tip_y_m, target_tip_z_m,
   target_R00, target_R10, target_R20,
   target_R01, target_R11, target_R21,
   target_gripper_distance_mm]

The rotation-6D representation is the first two columns of a 3×3 rotation matrix, flattened as [R[:,0], R[:,1]], not the first two rows.

For field-level storage, runtime decoding, timestamp fallback, and metadata definitions, see the complete schema.

Flange and gripper-tip coordinate frames

Teleoperation observation.state/action use flange poses in the robot-base frame. The corresponding *_gripper keys use the gripper-tip pose in the same robot-base frame. The published fixed transform is

T_base_gripper = T_base_flange @ T_flange_gripper

T_flange_gripper =
[[ 0.7660444431, -0.6427876097, 0, 0   ],
 [ 0.6427876097,  0.7660444431, 0, 0   ],
 [ 0,             0,            1, 0.2 ],
 [ 0,             0,            0, 1   ]]

Thus p_base_gripper = p_base_flange + R_base_flange @ [0,0,0.2] metres and R_base_gripper = R_base_flange @ Rz(+40°). For the reverse direction,

T_gripper_flange = inverse(T_flange_gripper) =
[[ 0.7660444431,  0.6427876097, 0,  0   ],
 [-0.6427876097,  0.7660444431, 0,  0   ],
 [ 0,             0,            1, -0.2 ],
 [ 0,             0,            0,  1   ]]

Translation is expressed in the flange frame; yaw is a right-handed local rotation about flange +Z. Gripper opening is copied without modification.

UMI is a separate channel: every TCP pose is expressed in the current episode's first-valid-TCP frame, T_local_i = inverse(T_0) @ T_raw_i, and its action is the next observed local TCP pose. Do not mix the two channels without respecting this frame convention.

Loading and visualization

Install a LeRobot version that supports dataset format v3 and the plotting dependencies:

python -m pip install "lerobot>=0.5" matplotlib numpy

Inspect all keys and their runtime shapes:

python examples/load_lerobot_dataset.py --repo-id qingzhu-robotics/TacRich-Manip-LeRobot-teleoperation-insert-peg-cylinder --episode-index 0

Render front/side/fisheye RGB, lossless uint16 depth, and both tactile maps:

python examples/visualize_episode.py \
  --repo-id qingzhu-robotics/TacRich-Manip-LeRobot-teleoperation-insert-peg-cylinder \
  --episode-index 0 \
  --frame-index 0 \
  --output episode0_frame0.png

LeRobot loaders normally expose RGB as CHW float tensors. Some torchvision versions expose 16-bit PNG depth as a signed int16 tensor containing the same bits; the provided visualizer safely reinterprets those bits as uint16 before plotting. Use the Parquet/Arrow value when exact float64 source timestamps are required, because a generic PyTorch scalar conversion can down-cast them.

File layout and metadata

README.md, LICENSE, CITATION.cff, AUTHORS.md
docs/DATASET_SCHEMA.md
examples/load_lerobot_dataset.py
examples/visualize_episode.py
meta/info.json
meta/stats.json
meta/tasks.parquet
meta/episodes/chunk-*/file-*.parquet
data/chunk-*/file-*.parquet
videos/<camera-key>/chunk-*/file-*.mp4

meta/info.json is the canonical feature/path declaration, meta/stats.json contains global statistics, meta/tasks.parquet maps task text to IDs, and meta/episodes/** maps every episode to frame and video ranges.

Depth-statistics note: In this release, LeRobot's generic image-statistics path recorded depth as three-channel normalized [0,1] image statistics. Those meta/stats.json depth values are not metric millimetre statistics and must not normalize uint16 depth. The stored PNG values and the provided visualizer remain exact.

Quality, provenance, and limitations

• —Conversion checks required fields, shapes, finite pose values, unit quaternions, task/episode counts, and every metadata-referenced shard.
• —Teleoperation actions come only from the migrated absolute arm_target_* columns. Raw action_delta_* is provenance and is never consumed by this converter. An action is a command at time t; servo latency means it is not expected to equal the next measured state exactly.
• —RGB video is lossy AV1; depth and low-dimensional fields are lossless apart from the documented float32 casts.
• —Sensor streams are asynchronous; use the source timestamps to measure age or alignment instead of assuming simultaneous exposure.
• —Tactile values are calibrated sensor responses, not force in newtons unless a separate force calibration is applied.
• —Real-robot trajectories may contain occlusion, lighting changes, contact transients, operator variation, and task failures. Review episodes before safety-critical use.

Intended use and safety

Intended uses include robot imitation learning, multimodal/tactile representation learning, contact-rich manipulation, synchronization research, and reproducible format conversion. The dataset does not constitute a safety controller or deployment guarantee. Validate workspace limits, action scaling, coordinate frames, and emergency-stop behavior on the target robot before any real-world execution.

License, attribution, and citation

This LeRobot dataset is distributed under the Apache License 2.0. Contributor and institutional attribution is in AUTHORS.md. Cite the dataset and record the exact Hugging Face commit revision used for experiments; machine-readable citation metadata is in CITATION.cff.

@dataset{tacrich_manip_insert_peg_cylinder_2026,
  author  = {{Qingzhu Robotics, TacRich-Manip Dataset Team}},
  title   = {{TacRich-Manip LeRobot v3: teleoperation/insert_peg_cylinder}},
  year    = {2026},
  version = {0.2.0},
  url     = {https://huggingface.co/datasets/qingzhu-robotics/TacRich-Manip-LeRobot-teleoperation-insert-peg-cylinder},
  note    = {Please also report the immutable Hub commit revision used.}
}

Format references

• —LeRobot Dataset v3 specification
• —Hugging Face dataset card guidance
• —LeRobot ALOHA insertion dataset