leesangoh/gr1-tabletop-augmented

GR-1 Tabletop, Augmented v2

Counterfactual action perturbations for the NVIDIA GR-1 tabletop manipulation dataset, with joint torque, fingertip force, and binary contact recorded alongside — and, for every perturbed rollout, the rendered future the perturbed action actually produces. Version 2 also supplies the complete recorded 10 Hz timeline re-rendered from the stored MuJoCo states through the same renderer, crop, resize, and JPEG path as the counterfactual futures.

The source dataset gives you what the robot did. This one also gives you what would have happened had it done something slightly different, and what the body felt while doing it.

What was added

The analytic baseline matters: subtracting it turns the learning problem from "predict the whole arm trajectory" into "predict what contact adds", which is where the residual actually lives. Measured on 767 anchors, the analytic rollout explains 0.97–0.99 of the variance in free space and −0.97 to −2.63 during contact — it is exact until something is touched.

Contents

data/perturbed/<task>/episode_XXXXXX.npz     counterfactual rollouts + rendered frames
data/recorded_rerender/<task>/episode_XXXXXX.npz
                                                canonical 10 Hz recorded-state renders
data/recorded_rerender/{meta.json,manifest.jsonl,source_hdf5_sha256.json}
                                                generation identity and source hashes
data/tactile_torque/                          per-demo torque and fingertip force
data/windows/                                 pre-cut training windows (history 8, horizon 16)
manifest_sha256.txt                           SHA-256 of all perturbed + recorded render archives

Every published file passed a full verification pass before upload: zip CRC over all bytes, exact key set, shape and dtype per array, all values finite, categories within {0,1,2}, contact binary and equal to force > 0.1, a frame offset table that starts at zero, increases, and ends exactly at the JPEG buffer length — and a decode of the first, middle, and last frame of each file.

Recorded re-render schema (v2)

There are 24,000 recorded episode archives containing 2,904,182 images. Only the odd 20 Hz frames are stored: archive row r is source frame 2r+1, which is the exact 10 Hz phase used by the action-conditioned predictor. These are direct renders of full recorded MuJoCo states; actions are not replayed and therefore cannot introduce trajectory drift.

The fixed image path is egoview 512x320 -> process_img_cotrain -> 256x256 -> JPEG quality 90. Cross-machine smoke outputs on RTX A6000, A5000, L40S, and RTX 6000 Ada were byte-identical. Before publication, every archive is checked for ZIP integrity, member schema, all per-image CRCs, JPEG decode, and correspondence with the source state/model hashes.

Episode file schema

Each .npz holds every rollout for one source episode. R is the rollout count (typically 93–97: ~24 anchors × 4 strengths, plus one unperturbed control per 24).

Frames are 256×256 RGB. The original MP4 remains the historical source observation; the v2 recorded re-render is the matched training domain for counterfactual prediction.

Reading a rollout

import io, numpy as np
from PIL import Image

d = np.load("episode_000003.npz", allow_pickle=False)
r = 0                                              # rollout index
npr = int(d["frames_per_rollout"])                 # 9
off = d["frames_offset"]
frames = [
    np.array(Image.open(io.BytesIO(
        d["frames_jpeg"][off[r*npr + i]: off[r*npr + i + 1]].tobytes())))
    for i in range(npr)
]                                                  # frames[0] conditions, [1:] are the future
residual = d["delta_q"][r]                         # (16, 39) what contact added
torque   = d["delta_tau"][r]                       # (16, 39)
touched  = d["contact"][r] > 0.5                   # (16, 10)

strength == 0 rollouts are unperturbed controls. They link a perturbed archive's renderer directly to the corresponding recorded state and are used as a timestamp/view integrity gate. The matched recorded re-renders remove the H.264-versus-JPEG source cue for model training.

Joint order

39 actuated DoF, in the order used by every array above:

waist 3            yaw, pitch, roll
right arm 7        shoulder pitch/roll/yaw, elbow pitch, wrist yaw/roll/pitch
right hand 11      thumb 3, index 2, middle 2, ring 2, pinky 2
left arm 7         (same as right)
left hand 11       (same as right)

Fingertip force and contact are 10-dimensional: 5 fingers × 2 hands.

Note that the action vector uses a different grouping (larm, lhand, rarm, rhand, waist) than the joint vector. They are not interchangeable without a permutation.

How the perturbations were made

Each anchor is replayed from the recorded state with the action sequence displaced along a random direction, scaled to the listed strength, and stepped through the same low-level controllers the source used. Physics and rendering come from one pass, so the frames and the proprioception describe the same rollout rather than two runs that happened to agree.

Directions are freshly drawn and the generation seed is not stored in the files, so the released set cannot be regenerated from the source trajectories alone. What fixes the identity of the rendered trajectories is manifest_sha256.txt at the repository root: one SHA-256 per perturbed or recorded render episode archive. Verify with

sha256sum -c manifest_sha256.txt

Generation ran across several machines. Where two of them independently produced the same episode, one copy was kept; each is an independently valid perturbation of the same source trajectory, and the manifest records which one was published.

Provenance and license

Derived from [nvidia/PhysicalAI-Robotics-GR00T-Teleop-Sim](https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-GR00T-Teleop-Sim) (NVIDIA GEAR), licensed CC BY-NC 4.0. Rendering uses robocasa-gr1-tabletop-tasks (MIT) and robosuite (MIT); the 3D assets appearing in rendered frames carry their own licenses (Lightwheel CC BY 4.0; Objaverse — mixed CC, including CC BY-NC-SA; Sketchfab — mixed).

This dataset is released under CC BY-NC-SA 4.0 — the most restrictive term inherited from its inputs. It is non-commercial, and adaptations must be shared alike.

Changes from the source: action perturbation and replay, analytic robot-only rollout, extraction of torque/fingertip force/contact, ego-view rendering of perturbed futures, state-exact re-rendering of the recorded 10 Hz timeline, and window pre-cutting. The source trajectories themselves are unmodified.

Version history

• —v2.0.0 (2026-09): adds the matched recorded-state JPEG timeline and generation/source provenance. Existing perturbed, tactile/torque, and window files are unchanged.
• —v1 (2026-08): initial counterfactual rollout release.

Citation

Please cite the source dataset alongside this one:

@misc{nvidia_gr1_teleop_sim,
  title  = {PhysicalAI-Robotics-GR00T-Teleop-Sim},
  author = {{NVIDIA GEAR}},
  year   = {2025},
  howpublished = {\url{https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-GR00T-Teleop-Sim}},
  note   = {CC BY-NC 4.0}
}