CoolFace
Datasetpublic

nvidia/PhysicalAI-Robotics-Locomanipulation-GRAIL

📢 News [2026-07-15] Released task-general tracking policy checkpoints trained on the released data. Follow the tracking doc to use them to track our released motion data. [2026-07-14] Updated data/pickup_table and data/pickup_ground. If you downloaded them before this date, please re-download. Dataset Overview Tabletop Pickup Ground Pickup Tabletop Manipulation Ground Manipulation Sitting Curb Slope… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-Locomanipulation-GRAIL.

sourceHugging Facecc-by-nc-4.0updated 16d agoView on Hugging Face
29likes30kdownloads
Dataset Card

📢 News

  • —[2026-07-15] Released task-general tracking policy checkpoints trained on the released data. Follow the tracking doc to use them to track our released motion data.
  • —[2026-07-14] Updated data/pickup_table and data/pickup_ground. If you downloaded them before this date, please re-download.

Dataset Overview

Tabletop PickupGround Pickup
<img src="assets/videos/pickup_table.gif" width="420"/><img src="assets/videos/pickup_ground.gif" width="420"/>
Tabletop ManipulationGround Manipulation
<img src="assets/videos/manip_tabletop.gif" width="420"/><img src="assets/videos/manip_large.gif" width="420"/>
SittingCurb
<img src="assets/videos/sitting.gif" width="420"/><img src="assets/videos/terrain_curbs.gif" width="420"/>
SlopeStairs
<img src="assets/videos/terrain_slopes.gif" width="420"/><img src="assets/videos/terrain_stairs.gif" width="420"/>

Description:

This dataset contains physics-validated 4D human-object interaction (HOI) trajectories for the Unitree G1 humanoid robot. It is generated by GRAIL (Generating Humanoid Loco-Manipulation from 3D Assets and Video Priors), an end-to-end pipeline that (1) acquires a 3D asset, (2) generates a synthetic character-object interaction video in Blender + Kling AI, (3) reconstructs the 4D HOI (SMPL-X human pose + object 6-DoF) from the video, (4) retargets the human motion to the G1 skeleton, and (5) drives a SONIC tracking policy in Isaac Lab — the released motion data is what the simulated G1 + object realize in simulation.

The release is partitioned by HOI category. Each motion ships with: the source synthetic video, the 4D HOI reconstruction (SMPL-X + object pose), the retargeted G1 robot trajectory, the post-RL object trajectory, and the object's USD asset (textures preserved).

The repo also ships the submodule checkpoints required to re-run the full GRAIL pipeline end-to-end (GEM-SMPL human pose estimation + FoundationPose object 6-DoF tracking + SONIC task general tracking). Note: the released SONIC checkpoints are intended to help reproduce our pipeline by fine-tuning the policies from these warm-up checkpoints as documented [here](https://nvlabs.github.io/GRAIL/tracking.html#finetuning-a-pick-up-policy). Releasing the final task-general tracking policy checkpoints for the released dataset is on our TODO list.

License/Terms of Use:

Use of the released dataset is governed by the Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0). The bundled checkpoints under checkpoint/ retain their respective upstream licenses:

  • —NVIDIA-produced (checkpoint/GEM-SMPL/outputs/, checkpoint/FoundationPose/, checkpoint/SONIC/) — NVIDIA Open Model License Agreement.
  • —Third-party (checkpoint/GEM-SMPL/inputs/: HMR2, ViTPose, VIMO, YOLOv8x, SMPL-X body model) — governed by the licenses of their respective upstream releases; please consult each upstream project before use.

Object USD assets under data/<hoi_category>/object_usd/ come from four asset sources, each with different downstream-license implications:

  • —RoboCasa-derived (pickup_table, pickup_ground) — derivative works of RoboCasa (UT Austin, NVIDIA), licensed under CC BY 4.0. Attribution to the RoboCasa Team is required when redistributing or building on these specific files.
  • —ComAsset-derived (advanced manipulation) — sourced from ComAsset (Kim et al., ECCV 2024), licensed under ODC-By v1.0. Attribution to the ComAsset authors is required. Individual meshes in ComAsset are originally collected from SketchFab and carry their own per-mesh licenses (listed in categories.json of the upstream ComAsset repo) — consult those before any redistribution, as some may impose restrictions beyond the CC BY-NC 4.0 terms that govern the rest of this dataset.
  • —Hunyuan3D-2.1 generated (advanced manipulation, stairs) — outputs of Hunyuan3D-2.1. Per the Tencent Hunyuan 3D 2.1 Community License, model outputs are user-owned and not subject to model-license restrictions, so these are released here under CC BY-NC 4.0. The Hunyuan3D-2.1 model itself (not used or redistributed by this dataset) carries territorial, MAU, and acceptable-use constraints — consult the upstream license if you intend to use the model.
  • —Procedurally generated (curb, slope, stairs) — GRAIL-original outputs, released under CC BY-NC 4.0.

The license: cc-by-nc-4.0 declared in the dataset metadata applies to the GRAIL-original outputs: motion trajectories, 4D HOI reconstructions, and per-motion metadata under data/<hoi_category>/{robot,objects,recon,meta}/, plus procedurally generated and Hunyuan3D-generated object assets. Bundled checkpoints under checkpoint/, RoboCasa-derived assets, and ComAsset-derived assets retain their respective upstream licenses as described above.

Use Case:

GRAIL is intended for use by individuals and professionals in fields such as robotics learning, machine learning, computer vision, and physics-based animation. Specific use cases include:

  • —4D HOI reconstruction — study 4D human-object interaction from a paired dataset of (synthetic video, SMPL-X recon, object 6-DoF, retargeted humanoid trajectory).
  • —Humanoid policy training — supervise RL or imitation-learning trackers on physically validated reference motions to learn whole-body loco-manipulation skills on the Unitree G1.
  • —Sim-to-real transfer — use the G1 trajectories directly as targets for a deployable controller, or as kinematic references for a learned residual policy.

Reference(s):

  • —Project page: <https://research.nvidia.com/labs/dair/grail/>
  • —Paper: <https://arxiv.org/abs/2606.05160>
  • —Code: <https://github.com/NVlabs/GRAIL>
  • —Documentation: <https://NVlabs.github.io/GRAIL/>

Dataset Layout:

nvidia/PhysicalAI-Robotics-Locomanipulation-GRAIL/
├── data/<hoi_category>/
│   ├── video/<hoi_category>__<object>__<NNN>.mp4       # source synthetic video
│   ├── recon/<hoi_category>__<object>__<NNN>.pkl       # 4D HOI recon (SMPL-X + object 6-DoF)
│   ├── robot/<hoi_category>__<object>__<NNN>.pkl       # post-RL G1 robot trajectory (29 body DOFs + hand_dof_pos)
│   ├── objects/<hoi_category>__<object>__<NNN>.pkl     # post-RL object 6-DoF trajectory
│   ├── meta/<hoi_category>__<object>__<NNN>.pkl        # per-motion metadata (lengths, contact flags, source IDs)
│   ├── object_usd/<hoi_category>__<object>__<NNN>.usd  # OpenUSD object asset
│   │   └── textures/<basename>/                        # per-USD texture subdir, refs rewritten in the USD
│   └── bps/<hoi_category>__<object>__<NNN>.npy         # BPS encoding of object 3D shape (only in pickup table/ground)
└── checkpoint/
    ├── GEM-SMPL/                                       # SMPL-X human pose estimation weights (HMR2, ViTPose, VIMO, YOLO, HMR4D)
    ├── FoundationPose/weights/                         # object 6-DoF estimator (refiner + scorer)
    └── SONIC/models/                                   # SONIC tracking checkpoints

The 3-digit NNN index restarts at 0 within each <object>.

Dataset Statistics per HOI Category:

CategoryWhat it is3D asset source# objects# motionsSeq. lengthTotal frames
pickup_tableTabletop pick-up — grasp an object from a table surface, lift, transportRoboCasa-derived 3D assets6852,99110 s747,750
pickup_groundGround pick-up — kneel to grasp an object from the floor and stand back upRoboCasa-derived 3D assets6441,73615 s651,000
sittingSitting — sit down on chair-like objectsHunyuan3D-generated 3D assets1891,7485 s218,500
slopeTerrain slope locomotion — walk over inclined surfacesProcedural terrain assets2001,88010 s470,000
curbTerrain curb locomotion — step over curb assetsProcedural terrain assets2001,76910 s442,250
stair (stair_p1 + stair_p2)Stair locomotion — ascend and descend stair assetsGenerated synthetic and real stair assets4,95212,18810 s3,047,000

Additional categories (tabletop / ground manipulation) are planned for subsequent releases.

Additional Statistics:

FieldValue
Synthetic video fps24 fps
Reconstructed motion fps24 fps
Released trajectory rate25 Hz
Reconstructed body modelSMPL-X (75 body DOFs + 45 × 2 hand DOFs)
Robot platformUnitree G1 (29 body DOFs + 7 × 2 hand DOFs)
Modalities per motionVideo (mp4), 4D HOI recon (pkl), robot traj (pkl), object traj (pkl), meta (pkl), object asset (USD + textures)

Data Visualization:

Released data can be rendered into kinematic-replay MP4s using GRAIL data visualization. The output can then be browsed using GRAIL web visualizer for hover-to-play previews.

bash
git clone https://github.com/NVlabs/GRAIL.git && cd GRAIL
# follow the installation guide to set up the `sonic` conda env
conda activate sonic
export DISPLAY=:1

# Batch — render up to 16 motions (default), defaults match the release (xyzw quats)
bash grail/visualization/scripts/visualize.sh \
    data/pickup_table

# Or render a single motion from its robot/<seq>.pkl
bash grail/visualization/scripts/visualize_single.sh \
    data/pickup_table/robot/pickup_table__apple_0__000.pkl

Outputs land alongside the motion library so the same directory works as input to the web visualizer:

data/<hoi_category>/vis/
├── <seq>.mp4                    one per motion
├── all_motions_combined.mp4     concat (only when --max_videos = 0)
└── examples_grid.mp4            4×4 or 2×2 grid (only when --max_videos = 0)

Knobs you may want to set:

Arg / envDefaultNotes
max_videos (2nd positional)16Cap on motions rendered. Pass 0 to render the whole library and produce the concat / grid MP4s.
cam_offset (3rd positional)1.5,-1.5,1.0Camera position relative to the motion centroid.

Data Collection Method:

Hybrid — Automatic. Each motion is the deterministic output of the GRAIL pipeline:

  1. 1.3D asset acquisition — RoboCasa-derived meshes, AI-generated meshes from Hunyuan3D-2.1, or procedural terrain assets. No real-world scans of identifiable objects.
  2. 2.2D HOI generation — a Blender rendering places a SMPL-X-rigged character with the object in an synthetic scene; a video exhibiting character-object interaction is generated through Kling AI.
  3. 3.4D HOI reconstruction — SMPL-X body pose recovered via GEM-SMPL (HMR2 + ViTPose + VIMO + HMR4D); object 6-DoF via FoundationPose conditioned on a SAM2 mask and a MoGe depth prior; jointly optimized in a multi-stage HOI optimizer.
  4. 4.Retargeting — SMPL-X human pose is retargeted to the Unitree G1 skeleton via the GMR IK + temporal-smoothing engine. Hand DOFs and per-motion USD assets are assembled in the same pass.
  5. 5.Task general tracking — the retargeted motion is used as a tracking reference for a SONIC policy in Isaac Lab. The post-RL object trajectory is the one realized by the simulated G1 + object under contact dynamics — guaranteed to be physically feasible by construction.

Disclaimer

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this dataset and the downstream models trained on it meet requirements for the relevant industry and use case and addresses unforeseen product misuse.

Ethical Considerations:

GRAIL trajectories are synthetic. No real individuals appear in the source videos, SMPL-X reconstructions, or any other modality — the entire pipeline is synthetic-character-only (the body model is parametric SMPL-X driven by retargeted character animation; no real-person mocap appears in the released motions). The 3D objects are AI-generated, procedurally generated, or licensed from synthetic asset libraries.

Users training policies on GRAIL are responsible for the safety properties of those policies once deployed on physical humanoids; the dataset itself is a kinematic reference and does not encode safety constraints, controller stability margins, or hardware torque/velocity envelopes. The Unitree G1 trajectories are guaranteed physically feasible in the Isaac Lab simulation environment under the SONIC tracker — sim-to-real transfer requires additional validation by the integrating team.

For more detailed information on ethical considerations for the upstream models GRAIL builds on, see the corresponding model cards: `nvidia/GEM-X` (human pose estimation) and the FoundationPose project page.

Explainability

FieldResponse
Intended Task/Domain:Humanoid whole-body loco-manipulation — supervising RL or imitation-learning controllers on physically validated reference motions for the Unitree G1.
Dataset Type:Trajectory dataset (per-motion robot + object 6-DoF + source video + SMPL-X recon).
Intended Users:Robotics learning researchers; machine-learning engineers; humanoid control researchers; computer-vision and graphics researchers working on HOI.
Output:Per-motion: G1 robot trajectory (T, 29) + hand DOFs, object 6-DoF (T, 7) (xyz + quat), input video (mp4), 4D HOI recon (SMPL-X parameters + object pose, world frame), per-motion metadata (meta/*.pkl), object asset (*.usd + textures).
Describe how the dataset was produced:A five-stage automated pipeline. (1) 3D asset acquisition; (2) character-object interaction rendered in Blender + video generation via Kling-AI; (3) 4D HOI reconstruction — SMPL-X via GEM-SMPL, object 6-DoF via FoundationPose; joint multi-stage optimizer; (4) retargeting to Unitree G1 via GMR; (5) task general tracking — the retargeted motion drives a SONIC policy in Isaac Lab, and the released robot/ and objects/ trajectories are what the simulated G1 + object actually realize under contact dynamics.
Name the adversely impacted groups this has been tested to deliver comparable outcomes regardless of:Not formally tested across demographic subgroups. Because the released trajectories are in G1 joint space rather than per-character body space, per-group outcome variation does not propagate to the dataset's primary downstream use (controller training).
Technical Limitations & Mitigation:(1) Single robot platform — G1 only; cross-embodiment retargeting requires additional work. (2) Synthetic-to-real domain gap — source videos are synthetic, so visual-feature-based downstream use (e.g. vision-conditioned policies) may need real-video fine-tuning. (3) No tactile / force annotations — the dataset is kinematic + 6-DoF only; contact forces are not exposed.
Verified to have met prescribed NVIDIA quality standards:Yes.
Performance Metrics:Per-motion physical feasibility is verified by construction (released motions are by definition those a trained SONIC tracker can follow).
Potential Known Risks:(1) Sim-to-real assumption mismatch — trajectories that succeed in Isaac Lab may not succeed on a physical G1 without additional residual learning, system-identification, or torque-limit checks. (2) Object-asset license inheritance — released USD assets and source videos inherit the license of their upstream source (RoboCasa, ComAsset, or Hunyuan3D); downstream users should confirm any application-specific redistribution constraints — see the License section above.
Licensing:The dataset itself is released under CC BY-NC 4.0 — attribution required, non-commercial use only. Bundled checkpoints, RoboCasa-derived object assets, and ComAsset-derived object assets retain their respective upstream licenses — see the License / Terms of Use section above for the full per-subtree breakdown.

Safety

FieldResponse
Dataset Application Field(s):Robotics Learning; Humanoid Whole-Body Control; Loco-Manipulation Research; Physics-Based Animation; Sim-to-Real Transfer Research.
Describe the life critical impact (if present).Not Applicable for direct dataset use. Downstream policies trained on GRAIL motions and deployed on a physical Unitree G1 are operating a hardware actuator and require independent safety validation by the integrating team (torque / velocity limit checks, emergency-stop integration, environment-specific risk assessment). The dataset itself does not encode hardware safety constraints.
Use Case Restrictions:Abide by the per-subtree licenses documented in the License / Terms of Use section above. In addition, GRAIL must not be used to: (1) train policies for deployment on humanoids in public-facing safety-critical roles (e.g. interacting with vulnerable populations) without independent safety validation; (2) produce or distribute deepfake content of real individuals — the dataset contains no real-person data, so any such use would require off-pipeline data that violates the upstream synthesis constraints; (3) violate the licenses of upstream third-party assets (object meshes, character libraries, motion sources).
Dataset restrictions:The Principle of Least Privilege (PoLP) is applied. Source assets (RoboCasa-derived 3D objects, synthetic character library, motion-source mocap) are tracked with NSpect IDs in the upstream pipelines. The release pipeline strips path-level provenance from the original cluster filesystems before publication.
Security considerations:The dataset is composed of pickled trajectories (Python .pkl), MP4 videos, OpenUSD assets, and a CSV manifest. Pickled files execute arbitrary Python on load; users should verify the integrity of downloaded files (HF hashes / commit signatures) and load them inside a trusted environment. The release ships no executable code, no model weights with auto-execute capability, and makes no network calls when loaded. Report security vulnerabilities to NVIDIA here.
Responsible AI practices:GRAIL is designed to advance public research on humanoid loco-manipulation. Users deploying derived policies on physical humanoids are responsible for hardware-side safety review (joint-limit / torque / E-stop / human-proximity safeguards) prior to deployment. NVIDIA encourages developers to implement validation harnesses, sim-to-real gap analysis, and conservative envelope checks in any pipeline that takes a GRAIL-trained policy to hardware.
nvidia/PhysicalAI-Robotics-Locomanipulation-GRAIL · CoolFace