aaronngx/failbench-robocasa-v2

FailBench RoboCasa v2 — contact-prediction dataset

Labeled robot-failure trials built on RoboCasa kitchen demos (PandaMobile / Franka). Each trial injects a hardware failure partway through a teleop/MimicGen demo, then records the contacts the failure causes during a 1-second settle. The supervised target is a 240×320 force-weighted contact heatmap in the agentview camera — the model learns to predict where a failure at a given pre-failure configuration will drive the robot/objects into contact.

Sibling dataset: `aaronngx/failbench-libero-v2` (identical schema, 240×320, LIBERO scenes). The two pool cleanly for cross-corpus training.

• —39,689 train trials across 20 task files (10 tasks × {human demos, MimicGen synthetic}).
• —Also ships a few held-out task files (TurnOffStove, TurnOnMicrowave, TurnOnStove_mg) for generalization eval — these are NOT in manifest_train.csv.
• —One HDF5 per task under v2/; per-trial groups at /trials/<trial_id>/.

⚠️ The one gotcha: import hdf5plugin first

Arrays are Blosc(lz4)-compressed (third-party HDF5 filter id 32001). Any reader must import hdf5plugin before h5py opens a file, or reads fail with "can't open plugin directory". (dataset.compression reports None for this filter — misleading; it IS compressed.) pip install hdf5plugin and import it first. The bundled load_failbench.py does this for you.

Quick start

pip install -r requirements.txt
python -c "from huggingface_hub import snapshot_download as s; \
  s('aaronngx/failbench-robocasa-v2', repo_type='dataset', local_dir='failbench-robocasa-v2')"

# load a trial + rebuild its contact target (pure h5py+numpy+scipy, no FailBench/MuJoCo):
python load_failbench.py --data_root failbench-robocasa-v2 \
                         --cache_root failbench-robocasa-v2/target_cache

Standalone load snippet

import load_failbench as fb                       # imports hdf5plugin for you
df = fb.load_manifest("failbench-robocasa-v2")     # the train manifest, as a DataFrame
row = df[df.n_contacts > 0].iloc[0]
trial = fb.read_trial(fb.resolve_h5("failbench-robocasa-v2", row.split, row.task), row.trial_id)

# (240,320) force-weighted contact heatmap = the prediction target:
target = fb.heatmap_from_projection("failbench-robocasa-v2/target_cache",
                                    row.split, row.task, row.trial_id)   # filtered (training)
# or rebuild from raw contacts + camera, no cache:  fb.heatmap_from_contacts(trial)

Repo layout

v2/<task>.h5                 # 22 task files (human; *_mg.h5 = MimicGen)
v2/manifest_train.csv        # 39,689 train trials (use THIS for training)
v2/manifest.csv              # full build incl. held-out + (pre-quarantine) rows
v2/quarantine.csv            # excluded trial ids
target_cache/robocasa/<task>.h5   # per-trial /<trial_id>/projection (N,3)[u,v,force] + failure_prob attr
load_failbench.py  requirements.txt  examples/quickstart.py
planner/  scripts/           # the exact FailBench load+train code subtree (see "Train")

Resolve files by `(split, task)` relative to your local root — the manifest's h5_path column holds the original build machine's absolute path and is not portable.

Per-trial schema

Each /trials/<trial_id>/ group. T=8 window @ 4 fps; settle S=50 steps (~1 s). Images are (H,W)=(240,320); depth is float16 metric.

Scalar attrs: trial_id, split(="robocasa"), task, demo_key, seed, fail_idx, traj_progress, failure_mode, failure_prob, is_holding.

Failure modes

GRIPPER_OPEN, SINGLE_JOINT, MULTI_JOINT, ALL_JOINTS, SLIPPERY_GRIP. Each trial samples one mode at a stratified traj_progress (failure fraction along the demo); failure_prob is the mode's prior, used as the per-trial heatmap weight.

Camera

The contact heatmap is projected into robot0_agentview_center, so the (240,320) target IS the prediction grid. For live eval, render that camera at 320×240 so recorded contacts line up with the prediction. (LIBERO uses agentview/eye_in_hand — same resolution/semantics.)

The target

target_cache/robocasa/<task>.h5[<trial_id>]/projection is (N,3)=[u,v,force_mag] of the failure-induced in-frame contacts (static/baseline resting contacts removed by a two-stage filter), plus a failure_prob attr. Rebuild the dense heatmap as scatter(force·failure_prob) → Gaussian blur(σ=4 px) (load_failbench.heatmap_from_projection, or planner.risk.v2_targets.build_target_from_projection on GPU). Pass log1p=True to match the GPU training path's mass compression.

Train (reproduce the in-repo benchmark model)

The bundled planner/ + scripts/benchmark/train_one.py is the exact load+train subtree (pure torch+torchvision; no MuJoCo). Targets are built on the fly — there is no --target_cache_root flag.

PYTHONPATH=. python -m scripts.benchmark.train_one \
    --robocasa_v2_root failbench-robocasa-v2/v2 \
    --model unet --modalities state rgb --T 8 \
    --split_by demo --epochs 10 --batch_size 64
# models: mlp | convdec | unet | transformer ;  modalities: state goal rgb depth failure_mode ...
# pooled cross-corpus: add --v2_root <libero v2 root> --splits libero_spatial libero_object libero_goal

License & attribution

Released under MIT. Built on RoboCasa (Nasiriany et al.), MimicGen (Mandlekar et al.), and robosuite — please cite those works. Contact labels and the failure-injection pipeline are from FailBench. Underlying demo content remains under its upstream RoboCasa license.