irl-kit/IA-Bench

IA-bench ( Interaction-Aware Bench)

Human ground-truth annotations of the interacted object for robot manipulation subtasks. Each sample is one subtask: the full subtask video clip, the gripper proprioception aligned 1:1 to those frames, the language instruction, and two boxes: initial_object_box (object on the first frame) and target_object_box (object on the last frame). Boxes are pixel [x1, y1, x2, y2].

Configs

from datasets import load_dataset
ds = load_dataset("irl-kit/IA-bench", "bridge_lerobot", split="validation")
ds = load_dataset("irl-kit/IA-bench", "all", split="test")

The source repository stores VideoFolder metadata as metadata.jsonl, which is loaded automatically by load_dataset. The all config contains 1,880 unique samples. Hugging Face's headline row total sums overlapping configs and therefore does not represent the number of unique samples.

Per-frame fields

• —video: the full subtask clip (mp4, native resolution, all frames).
• —native_fps / effective_fps: source fps and the clip fps (equal unless a frame cap is applied). frame_indices gives original frame indices (timestamp = index / native_fps); proprio is aligned 1:1 with the frames.
• —proprio (JSON, decode with json.loads): unified, raw (un-normalized) per-frame proprioception. Consistent field names across datasets: gripper_state (open/close, inverted to a common convention), eef_state (6/7-dim end-effector pose), joint_pos (arm joint positions). Bimanual datasets prefix fields with left_/right_. proprio_dims gives the per-field feature dimension; proprio_keys lists the present fields. Availability/dim varies by embodiment (e.g. OXE exposes gripper only; AgiBot eef is xyz-only).

Evaluation

eval_ia_bench.py scores predicted start (initial_object_box) and target (target_object_box) boxes against the GT and reports the paper metrics: acc@IoU, AUROC, AURC, E-AURC, cov@90, cov@95, R@90, R@95 (per dataset + overall). A prediction row needs dataset, trajectory_name, subtask_index, the two boxes, and a confidence score.

python eval_ia_bench.py --predictions preds.jsonl --gt-repo irl-kit/IA-bench

Citation

If you use IA-bench, please cite:

@misc{blank2026sparcreliablespatialannotations,
      title={SPARC: Reliable Spatial Annotations from Robot Demonstrations at Scale}, 
      author={Nils Blank and Paul Mattes and Maximilian Xiling Li and Jakub Suliga and Thomas Roth and Moritz Reuss and Pankhuri Vanjani and Rudolf Lioutikov},
      year={2026},
      eprint={2606.13497},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2606.13497}, 
}