leesangoh/dynamics-probing

PhysProbe Dynamics Probing Dataset

Manipulation episodes from Isaac Lab collected for probing physics understanding in video world models (V-JEPA 2, VideoMAE, DINOv2). Each episode includes dual-camera RGB (384×384), robot state, scripted/RL actions, and per-timestep physics ground truth (contact forces, object kinematics, physics randomization parameters).

Tasks

Format

LeRobot V2 layout. Per-task:

<task>/
  data/chunk-000/episode_NNNNNN.parquet
  videos/chunk-000/observation.images.image_0/episode_NNNNNN.mp4   # table_cam
  videos/chunk-000/observation.images.image_1/episode_NNNNNN.mp4   # wrist_cam
  meta/info.json
  meta/episodes.jsonl
  meta/tasks.jsonl
  meta/stats.json
  meta/modality.json

Per-episode parquet columns:

• —observation.state (8D): 7 joint positions + 1 gripper
• —action (3–8D, task-dependent)
• —next.reward, next.done
• —physics_gt.* (task-specific — see below)
• —Frame index, timestep, episode index metadata

Physics Ground Truth (physics_gt.*)

Common across all tasks

• —ee_position (3), ee_orientation (4), ee_velocity (3), ee_angular_velocity (3) — end-effector kinematics
• —joint_pos (7), joint_vel (7) — arm joint state
• —phase (1) — task phase label (task-dependent enum; 7 = idle)

Contact fields (per task)

Push, Strike, Reach:

• —contact_flag (1), contact_force (3), contact_point (3) — aggregate ee↔object + object↔surface
• —contact_finger_l_object_flag/force, contact_finger_r_object_flag/force — per-finger ee↔object (new in 2026-04-23)
• —contact_object_surface_flag/force — object↔surface (new in 2026-04-23)

PegInsert, NutThread:

• —contact_flag (1), contact_force (3), contact_point (3) — aggregate
• —contact_finger_l_peg_flag/force, contact_finger_r_peg_flag/force (PegInsert) — finger↔peg
• —contact_finger_l_nut_flag/force, contact_finger_r_nut_flag/force (NutThread) — finger↔nut
• —contact_peg_socket_flag/force (PegInsert) — peg↔hole (reconstructed as residual from peg total contact minus finger reactions; direct pair filter unsupported in Factory direct env)
• —contact_nut_bolt_flag/force (NutThread) — nut↔bolt (direct exact-pair sensor)

Drawer:

• —handle_position (3), handle_velocity (3)
• —drawer_joint_pos (1), drawer_joint_vel (1)

Task-specific kinematics

Push/Strike/Reach (Step 0):

• —object_position (3), object_orientation (4), object_velocity (3), object_angular_velocity (3)
• —target_position (3) — placeholder [0, 0, 0] (no target in Step 0)

PegInsert/NutThread:

• —held_position (3), held_orientation (4), held_velocity (3), held_angular_velocity (3) — peg/nut kinematics
• —fixed_position (3), fixed_orientation (4) — hole/bolt pose
• —insertion_depth (1) — peg_insert only

Physics randomization parameters (per episode)

Stored in episode metadata (physics_gt.*_{static,dynamic}_friction, *_mass, etc.) — see per-task schema above for exact fields.

2026-04-23 Recollection Note

The previous version of this dataset (before 2026-04-23) had a data-collection bug: contact forces were zero-filled across all tasks because the sensor configuration did not use the proper body filters / the Factory direct env does not support get_net_contact_forces on ArticulationView. This version fixes the following:

1. 1.Push, Strike, Drawer, Reach: Per-pair ContactSensor with filter_prim_paths_expr on finger/object bodies → real nonzero contact forces.
2. 2.NutThread: Direct exact-pair sensor (contact_nut_bolt_*) → direct nut↔bolt force.
3. 3.PegInsert: GPU pair filtering on hole is unsupported in direct Factory env. Peg↔socket contact is reconstructed as a residual: F_peg_socket = F_peg_total - F_finger_l_peg - F_finger_r_peg (Newton's 3rd law). This is sparser and noisier than a direct sensor; finger-grip force dominates and is subtracted, so pay attention when using contact_peg_socket_* for downstream probing.
4. 4.Phase label: Now correctly tracks scripted policy state transitions (previously always 7/idle for RL policies).

Known caveats (unchanged from previous release)

• —Push/Strike are Step 0: no target, success=True for all, target_position=[0,0,0].
• —Drawer randomizes only drawer_joint_damping (Isaac Lab env limitation — handle friction/mass are fixed).
• —Factory tasks (PegInsert, NutThread): collection uses scripted policy; success rate is low for unguided inserts.

Collection Pipeline

• —Isaac Sim 4.5 / Isaac Lab v2.2.1
• —Scripted oracle policies + optional RL checkpoints (rl_games for Factory, rsl_rl for Drawer)
• —Dual camera rendering at 384×384, 15 fps
• —num_envs per task: 8 (Factory), 16 (others) parallel
• —Hardware: 4× NVIDIA A6000 48 GB

Collection script: Leesangoh/PhysREPA_Tasks (archive_data_collection/collect_sample_data.py).

Intended use

This dataset is designed for probing physics understanding in pretrained video encoders:

• —Linear probes from mean-pooled features onto physics_gt.* targets
• —Temporal-aligned window-level probing (per-window features → per-window targets)
• —Do NOT episode-mean aggregate features/targets for kinematic probing — that collapses temporal structure and produces misleading results.

Citation

TBD (paper in preparation).