junhee1998/gr00t-n15-robocasa-gr1-eval

GR00T N1.5 on RoboCasa GR-1 Tabletop — Evaluation Trajectories

Per-simulator-step recordings of 1,200 evaluation episodes (24 tasks × 50 episodes) of NVIDIA's GR00T N1.5 vision-language-action model on the RoboCasa GR-1 Tabletop Tasks benchmark.

Each episode stores every low-level transition — ego-view frames, robot state, executed actions, rewards, full MuJoCo state, and 3D poses of every object and scene body — so that rollouts can be re-analyzed or re-rendered without re-running the policy.

Episodes are split into success/ and fail/ by the success field of each episode's metadata.json.

Results obtained in this run

At n=50 the per-task binomial standard error is ±7.1 pp, so the 24-task mean has SE ±1.4 pp. The −2.6 pp difference is within noise, and only 4 of 24 tasks differ by more than 2σ (±14 pp), scattered in both directions (+30, −16, −16, −18 pp). This run reproduces the published GR00T N1.5 numbers.

<details> <summary>Per-task success rate vs. the published table</summary>

Published numbers from examples/RoboCasa/README.md in NVIDIA/Isaac-GR00T (50 rollouts each). </details>

What the failures look like

All 653 failures ended by timeout; none terminated on an incorrect-action judgment. Classifying each failure by the stage it stalled at (from the 3D trajectories of the hand and the target object):

Grasping is the dominant failure mode: the hand gets within 11.8 cm of the target (successful episodes: 10.5 cm) but the object moves only 9 mm.

Signals that would indicate a reasoning rather than control failure are rare: the grasp_distractor_obj flag never fires (0 / 1,200); the hand ends up closer to a non-target object in 22 / 653 failures (3.4%); the target is never inside the ego-view frame in 7 / 653 (1.1%), and never in a successful episode.

Evaluation setup

Only arms, hands and waist are actuated; legs and neck are held at zero and are not recorded.

Layout

robocasa_gr1_tabletop_tasks/gr1_unified/
└── <TASK>_GR1ArmsAndWaistFourierHands_Env/
    └── run-<timestamp>-<uuid>/
        ├── success/episode-XXXXXX/
        └── fail/episode-XXXXXX/
            ├── trajectory.hdf5    # every simulator step
            ├── metadata.json      # episode summary + scene composition
            ├── model.xml          # full MuJoCo scene
            └── preview.mp4        # ego-view preview

episode_index.csv at the repo root lists every episode with its task, outcome and step count.

trajectory.hdf5

N = number of steps. Observations and physics state have `N+1` rows, actions and rewards have `N` (the reset state is stored first), so observations[t] --actions[t]--> observations[t+1].

Object roles: obj = target, container = destination, obj_container = source, distractor_* = distractors.

model.xml — asset paths are placeholders

Mesh and texture references are stored as {ROBOCASA_ROOT}/... and {ROBOSUITE_ROOT}/.... Substitute your local checkout paths before loading:

xml = open('episode-000000/model.xml').read() \
        .replace('{ROBOCASA_ROOT}', '/path/to/robocasa-gr1-tabletop-tasks') \
        .replace('{ROBOSUITE_ROOT}', '/path/to/robosuite')

Quick start

import h5py, json

ep = 'robocasa_gr1_tabletop_tasks/gr1_unified/<TASK>/run-.../success/episode-000026'
meta = json.load(open(f'{ep}/metadata.json'))
print(meta['episode_metadata']['lang'], meta['success'], meta['num_steps'])

with h5py.File(f'{ep}/trajectory.hdf5') as f:
    frames = f['observations/video.ego_view_bg_crop_pad_res256_freq20']   # (N+1,256,256,3)
    names  = [n.decode() for n in f['objects/names'][:]]
    target = f['objects/positions'][:, names.index('obj'), :]             # (N+1,3)

Files are HDF5, so the Hub dataset viewer does not preview them. Use hf download or huggingface_hub.snapshot_download; --include lets you fetch a single task.

Size

105 GB total; ~97% is the two ego-view frame arrays inside trajectory.hdf5 (median 90 MB per episode, max 168 MB). Failures average 126 MB and successes 41 MB, because every failure ran to its 720-step limit.

Related

A companion dataset records the same 24 tasks under the same simulator, client and machine for the ZR-0 VLA: `junhee1998/zr0-robocasa-gr1-eval`. That run reached 33.0% against a reference of 69.3%. Since GR00T N1.5 reproduces its own reference numbers on this same setup, the ZR-0 gap is not attributable to a shared evaluation-pipeline fault.

Limitations

• —Every failure ended by timeout; no episode terminated on an incorrect-action judgment.
• —Single run per task; no seed variation. 50 episodes gives ±7.1 pp per task.
• —grasp_distractor_obj never fires, yet episodes exist where the hand demonstrably manipulates a distractor. Read that flag as "the simulator's distractor-grasp condition was never met", not as "no distractor was ever handled".

License and attribution

Released under CC BY-NC 4.0. The recordings contain renders of 3D assets from RoboCasa, some of which carry NonCommercial terms — hence the NC clause.

Assets appearing in these episodes (344 distinct assets):

• —Objaverse (267) — dataset under ODC-By 1.0; individual objects under CC-BY 4.0 / CC-BY-NC 4.0 / CC-BY-NC-SA 4.0 / CC-BY-SA 4.0 / CC0 1.0.
• —Lightwheel (71) — CC BY 4.0, © Lightwheel.
• —Sketchfab (6) — CC-BY / CC-BY-NC; per-asset credits in `ASSETS_ATTRIBUTIONS.md`.

Upstream components: RoboCasa GR-1 Tabletop Tasks (MIT, © NVIDIA GEAR), robosuite (MIT), Isaac-GR00T (Apache-2.0, © NVIDIA). The checkpoint was post-trained by its authors on NVIDIA PhysicalAI-Robotics-GR00T-Teleop-Sim.

If you are an asset author and want a credit corrected or content removed, please open a discussion on this repository.

Citation

@misc{gr00t_n15_robocasa_gr1_eval_2026,
  title  = {GR00T N1.5 on RoboCasa GR-1 Tabletop: Evaluation Trajectories},
  author = {Junhee},
  year   = {2026},
  url    = {https://huggingface.co/datasets/junhee1998/gr00t-n15-robocasa-gr1-eval}
}