junhee1998/zr0-robocasa-gr1-eval

ZR-0 on RoboCasa GR-1 Tabletop — Evaluation Trajectories

Per-simulator-step recordings of 2,400 evaluation episodes (24 tasks × 100 episodes) of the ZR-0 vision-language-action model on the RoboCasa GR-1 Tabletop Tasks benchmark.

Each episode stores every low-level transition — ego-view frames, robot state, executed actions, rewards, full MuJoCo state, and 3D poses of every object and scene body — so that rollouts can be re-analyzed or re-rendered without re-running the policy.

Episodes are split into success/ and fail/ by the success field of each episode's metadata.json.

Results obtained in this run

[!IMPORTANT] These success rates are substantially below the numbers reported by the ZR-0 authors (69.3% mean over the same 24 tasks). All 24 tasks are lower, by 15–59 percentage points. The cause of this gap has not been determined. Do not read these numbers as a measurement of ZR-0's capability. Specifically untested at the time of release: - All rollouts used --inference_mode direct_action, which skips embodied chain-of-thought entirely. The reference numbers may correspond to subtask_then_action. - Four policy servers were run concurrently on a single GPU. This dataset is published as evaluation traces for failure analysis, not as a benchmark result.

Evaluation setup

Only arms, hands and waist are actuated; legs and neck are held at zero and are not recorded.

Layout

robocasa_gr1_tabletop_tasks/gr1_unified/
└── <TASK>_GR1ArmsAndWaistFourierHands_Env/
    └── run-<timestamp>-<uuid>/
        ├── success/episode-XXXXXX/
        └── fail/episode-XXXXXX/
            ├── trajectory.hdf5    # every simulator step
            ├── metadata.json      # episode summary + scene composition
            ├── model.xml          # full MuJoCo scene
            └── preview.mp4        # ego-view preview

episode_index.csv at the repo root lists every episode with its task, outcome and step count.

trajectory.hdf5

N = number of steps. Observations and physics state have `N+1` rows, actions and rewards have `N` (the reset state is stored first), so observations[t] --actions[t]--> observations[t+1].

Object roles: obj = target, container = destination, obj_container = source, distractor_* = distractors.

model.xml — asset paths are placeholders

Mesh and texture references are stored as {ROBOCASA_ROOT}/... and {ROBOSUITE_ROOT}/.... Substitute your local checkout paths before loading:

xml = open('episode-000000/model.xml').read() \
        .replace('{ROBOCASA_ROOT}', '/path/to/robocasa-gr1-tabletop-tasks') \
        .replace('{ROBOSUITE_ROOT}', '/path/to/robosuite')

Quick start

import h5py, json

ep = 'robocasa_gr1_tabletop_tasks/gr1_unified/<TASK>/run-.../success/episode-000026'
meta = json.load(open(f'{ep}/metadata.json'))
print(meta['episode_metadata']['lang'], meta['success'], meta['num_steps'])

with h5py.File(f'{ep}/trajectory.hdf5') as f:
    frames = f['observations/video.ego_view_bg_crop_pad_res256_freq20']   # (N+1,256,256,3)
    names  = [n.decode() for n in f['objects/names'][:]]
    target = f['objects/positions'][:, names.index('obj'), :]             # (N+1,3)

Files are HDF5, so the Hub dataset viewer does not preview them. Use hf download or huggingface_hub.snapshot_download; --include lets you fetch a single task.

Size

312 GB total; ~97% is the two ego-view frame arrays inside trajectory.hdf5 (median 58 MB per episode, max 164 MB). Everything except the frames is ~9 GB. Failures are larger than successes because every failure ran to its step limit.

Limitations

• —Every failure ended by timeout; no episode terminated on an incorrect-action judgment.
• —4 of 24 tasks have ≤1 success, so per-task statistics there are unstable.
• —The gap against the reference numbers is unexplained (see above).
• —Single run per task; no seed variation.

License and attribution

Released under CC BY-NC 4.0. The recordings contain renders of 3D assets from RoboCasa, some of which carry NonCommercial terms — hence the NC clause.

Assets appearing in these episodes (344 distinct assets):

• —Objaverse (267) — dataset under ODC-By 1.0; individual objects under CC-BY 4.0 / CC-BY-NC 4.0 / CC-BY-NC-SA 4.0 / CC-BY-SA 4.0 / CC0 1.0.
• —Lightwheel (71) — CC BY 4.0, © Lightwheel.
• —Sketchfab (6) — CC-BY / CC-BY-NC; per-asset credits in `ASSETS_ATTRIBUTIONS.md`.

Upstream components: RoboCasa GR-1 Tabletop Tasks (MIT, © NVIDIA GEAR), robosuite (MIT), ZR-0 (Apache-2.0). The ZR-0 checkpoint was fine-tuned by its authors on NVIDIA PhysicalAI-Robotics-GR00T-Teleop-Sim.

If you are an asset author and want a credit corrected or content removed, please open a discussion on this repository.

Citation

@misc{zr0_robocasa_gr1_eval_2026,
  title  = {ZR-0 on RoboCasa GR-1 Tabletop: Evaluation Trajectories},
  author = {Junhee},
  year   = {2026},
  url    = {https://huggingface.co/datasets/junhee1998/zr0-robocasa-gr1-eval}
}