ckwolfe/benchmarks-viz-tiles

Dexterous Manipulation Benchmark — Cross-Method Evaluation

Cross-method evaluation scaffolding for dexterous manipulation, comparing 4 methods × 4 hands × 2 datasets × 7 trajectories on identical hardware, scenes, and reference trajectories.

Tag: `v0.7-mt-complete` · Trained checkpoints in the private companion repo `ckwolfe/benchmarks-trained-ckpts`.

TL;DR

Results — averaged across all real eval cells

All numbers are per-cell means; full per-(method,hand,traj) rows live in metrics/*.jsonl.

Interpretation notes for reviewers

• —DM "5 seeds" = 5 training reps, shared RL seed=42. All 140 public DM checkpoints in staging/run_paths.json were trained with seed: 42 per their config.json. The 5-rep variance reflects rl_games's env-step + mini-epoch stochasticity, not a seed sweep.
• —Open-loop vs closed-loop. Spider (sampling) and Oracle (kinematic replay) are fundamentally different from MT/DM (closed-loop learned controllers). Reading them as one leaderboard is misleading. Paper Table 1 splits into two sub-tables keyed on method class.
• —Simulator confound. MT runs in IsaacGym, DM in Genesis, Spider in MuJoCo-Warp, Oracle in MuJoCo. Contact model, integrator, and timestep differ. Tracking-err is most comparable within-class; cross-class comparisons are illustrative only.
• —Divergent-rollout guard. Rows where the rollout diverged (tracking_err_mean > tracking_err_max + 0.01 or add_mean > 5 m) are blanked to null at write time. Schema validator in shared/bench/schema.py enforces this on append.

Datasets × hands

Per-cell coverage (✓ real eval, · missing)

ManipTrans × OakInk | | liftboard | pickspoon | pourtube | stirbeaker | uncap | unplug | wipe_board | |---|---|---|---|---|---|---|---| | allegro | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | inspire | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | schunk | · | · | · | · | · | · | · | | xhand | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |

Schunk retrain running (epoch ~550/3000, reward climbing). Arrival: `v0.8-schunk-retrained`.

DexMachina × Arctic | | ketchup30 | box30 | mixer30 | ketchup40 | mixer40 | notebook40 | waffleiron40 | |---|---|---|---|---|---|---|---| | allegro | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | inspire | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | schunk | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | xhand | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |

Standardized videos

All videos are 720×480 @ 30 fps h264 crf 20, rendered through a shared MuJoCo scene with BENCH_CAMERA["front"] (pos=[0,-1.6,2.2], lookat=[0,-0.1,1.2], fov=30°). Files live under videos_std/.

Native (per-method) videos

Also published for cross-check: each method's own renderer (different camera, resolution, overlays). Under videos/:

• —dexmachina_* (Genesis camera): 16 oakink demo-playback clips
• —spider_* (MJX camera): 26 oakink cells
• —ours_* (MuJoCo camera): 9 oakink cells

How to reproduce one cell

# Clone the benchmarking repo (not shipped here — the datasets repo only hosts artifacts).
git clone https://github.com/<you>/benchmarking && cd benchmarking
./scripts/run.sh maniptrans --hand xhand --dataset oakink_v2 --traj lift_board_bimanual --seed 42 --viz none --dev
# → appends MetricsRow to outputs/metrics/maniptrans.jsonl and (via bench_hooks.patch)
#   dumps qpos to outputs/mt_qpos/<run_id>.npz for post-hoc video rendering.

MUJOCO_GL=egl python scripts/render_standardized.py --source mt --hand xhand --traj lift_board
# → writes videos_std/mt_xhand_oakink_v2_lift_board_bimanual_seed42_default.mp4

Full sweep

./scripts/deploy_all.sh               # all 4 methods end-to-end, ~14 GPU-hr

Per-method deploy scripts under scripts/deploy_{spider,dexmachina,maniptrans,ours}.sh.

Files in this repo

metrics/
  maniptrans.jsonl     # 613 rows (27 real)
  dexmachina.jsonl     # 229 rows (225 real)
  spider.jsonl         # 293 rows (28 real — 1 divergent row stripped)
  ours.jsonl           # 20 rows (20 real)
videos_std/
  std_*.mp4            # 28 Oracle kinematic replays (oakink)
  mt_*.mp4             # 21 ManipTrans captured-qpos replays (oakink)
videos/
  dexmachina_*.mp4     # 16 DM native (oakink demo playback)
  spider_*.mp4         # 26 Spider native (MJX camera)
  ours_*.mp4           # 9 Oracle native (MuJoCo camera)
scripts/
  render_standardized.py   # MJ renderer shared across methods
patches/
  mt_bench_hooks.patch      # MT qpos-capture upstream patch (253 lines)
  mt_trajs_configs/         # 7 oakink YAMLs with upstream-compatible data_idx
STATUS.md                   # live coverage snapshot

Metric schema (shared/bench/schema.py::MetricsRow)

Validator (at row-append time) rejects tracking_err_mean > tracking_err_max + 0.01 and add_mean > 5 m — those are the known divergent-rollout failure modes.

Limitations & what's not here

Citation

Paper in preparation. If you reference this artifact in the meantime, please cite:

@misc{benchmarks-viz-tiles-2026,
  title  = {Dexterous Manipulation Benchmark — Cross-Method Evaluation Tiles},
  author = {C.K. Wolfe and T. Sadjadpour},
  year   = {2026},
  url    = {https://huggingface.co/datasets/ckwolfe/benchmarks-viz-tiles},
  note   = {Tag v0.7-mt-complete}
}

Upstream methods retain their own licenses/attributions. See:

• —ManipTrans — ManipTrans/ManipTrans; checkpoints: `LiKailin/ManipTrans`
• —DexMachina — MandiZhao/dexmachina; 140 RL ckpts via upstream GDrive
• —Spider — facebookresearch/spider; retargets: `retarget/retarget_example`