shinben0327/hoi-retarget

HOI-Retarget — Humanoid Human–Object Interaction Motion

Project page · Code · 3D viewer · Paper: manuscript under review

Learning from demonstration (LfD) has enabled humanoid robots to acquire diverse whole-body skills, but extending this paradigm to human-object interaction (HOI) is limited by the availability of robot-compatible interaction references. We present HOI-Retarget, a contact-centric retargeting method that transfers HOI onto a humanoid robot for large-scale motion-data generation. Its windowed trajectory optimization uses every labeled contact as a target in the object frame, balancing body tracking, foot support and smoothness under the robot's kinematic limits. The method can augment a single demonstration across object sizes, absorb contacts reconstructed from monocular video, and extend to several robots manipulating one object.

This is the corpus that method produces: 13,904 humanoid trajectories from five human–object interaction capture sets. Each row is one (motion, robot) pair — joint angles, a floating base, the object's 6-DoF pose, and the per-link contact flags that say which palms and feet touch the object at every frame.

Contents

Per dataset (G1, object_scale 0.83)

Per robot

The H2 is taller and keeps less: 90.9 % on OMOMO, 97.9 % on ParaHome, 93.3 % on NeuralDome, 78.7 % on CoRoleHOI, 96.1 % on IMHD².

Object meshes

The 13 OMOMO objects ship under assets/objects/ (44 MB of URDF, mesh and surface samples), so the 4,421 OMOMO motions are usable as downloaded. Point the code at this directory:

export HOI_RETARGET_OBJECT_ROOTS=<this download>

The variable adds search roots; do not use HOI_RETARGET_ASSETS, which replaces the whole asset root and hides the robot models.

The other four datasets' meshes reach you through InterAct under CC BY-NC-SA 4.0 with written authorisation and are not redistributed here. Place them under the same root in the layout object_model_path names:

<root>/assets/objects/<object>.urdf       OMOMO — included above
<root>/<dataset>/_assets/<object>.urdf    ParaHome, NeuralDome, CoRoleHOI, IMHD²

`docs/DATA.md` covers obtaining them, and hoi-retarget-stage-object writes the URDF and surface samples for a mesh of your own.

Object names repeat across datasets but the meshes do not, so objects are keyed <dataset>__<object>. `docs/objects.jpg` is a contact sheet of all 75.

Quick start

pip install datasets

from datasets import load_dataset
import numpy as np

ds = load_dataset("shinben0327/hoi-retarget", split="train")           # everything
ds = load_dataset("shinben0327/hoi-retarget", "omomo", split="train")  # one source dataset

clean = ds.filter(lambda r: r["qc_pass"])          # the curated subset
g1    = ds.filter(lambda r: r["robot"] == "unitree_g1")

r = clean[0]
dof = np.array([np.asarray(x) for x in r["dof_pos"]])    # (T, 29) joint angles, radians
obj = np.array([np.asarray(x) for x in r["object_pos"]]) # (T, 3) object position, metres
print(r["clip_id"], dof.shape, r["object"], r["qc_flags"])

Every column is described in `docs/SCHEMA.md`.

The video column ships undecoded, so datasets alone is enough: each cell is {"bytes": <mp4>, "path": <name>}. For decoded frames, install torchcodec and ds.cast_column("video", Video()).

examples/to_pkl.py converts a row back into the contact_window.pkl layout that hoi-retarget --mode contact writes, so a row drops straight into the code repository for re-solving, rendering or contact editing.

Quality control

qc_pass is evaluated per (motion, robot): the two robots have different joint limits and fail differently.

qc_pass = False  if  wrist_runfrac > 0.50            # wrist_sustained: a wrist joint pinned beyond
                                                     # 80 % of its own half-range for >50 % of the clip
                 or  foldx_p80 >= 50                 # body_folded: the trunk folds 50 deg further
                                                     # from vertical than the human's did
                 or (limit_sat_pct >= 15 and foldx_p80 >= 15)
                                                     # body_folded: joints against their stops,
                                                     # corroborated by a real fold

foldx_p80 is source-relative — the robot's trunk fold minus the human's — so a person who genuinely squats or sits scores near zero. The flags catch physically implausible motion, not unusual interaction: a clip where the robot kicks a box or never uses its hands passes. Failing rows ship flagged, and qc_flags names the rule that fired (wrist_sustained, body_folded).

Of the three inputs only wrist_runfrac is a column here, so qc_pass can be read but not recomputed from this dataset alone.

Seeing the motions

Every row carries a 320 × 320 render of that trajectory in the video column (~50 kB), so the table preview plays each clip in place.

HOI-Retarget Contact Playback plays any clip in 3D in the browser, with search and filtering by dataset, robot, QC result or collaborative pair. It reads viewer/, which holds the same trajectories as little-endian float16, one file per (motion, robot):

viewer/index.json                  every clip, with its agents and metadata (0.25 MB gzipped)
viewer/shards/<ds>__<robot>.json   the same records grouped, for lazy loading
viewer/bin/<ds>__<robot>/<subject>.bin

Each .bin is block-major — each array whole, one after another, not interleaved per frame:

body_pos   (T, NB, 3)   then  body_quat  (T, NB, 4)  wxyz  then
object_pos (T, 3)       then  object_rot (T, 4)      wxyz  then
contact    (T, NC)      0.0 / 1.0

so object_pos begins at half-offset T*NB*3 + T*NB*4. NB is 42 for the G1 and 36 for the H2; NC is 4. Slice by block, not by a per-frame stride: the file is the same size either way, so a length check cannot tell the two readings apart.

Collaborative clips carry two agents sharing one object. Both agents store their own copy of the object track; they agree once each agent's z_offset_m is applied, and a viewer should draw the object once.

Licensing

CC BY-NC-SA 4.0: non-commercial, attribution, and derivatives carry the same licence. That is the most restrictive term among the sources, and it propagates.

These trajectories are derivative works of the source motion capture, and the contact annotations they were optimised against originate with InterAct under CC BY-NC-SA 4.0. Cite the source dataset for any clip you use; the dataset column names it and NOTICE.md gives the reference.

The OMOMO object meshes are redistributed from InterMimic under MIT (geometry © 2023 Jiaman Li, assets © 2025 Sirui Xu), notice in assets/objects/LICENSE-OBJECTS. Robot models are Unitree's, BSD-3-Clause (`docs/LICENSE-unitree.txt`). The retargeting code is BSD-3-Clause.

Citation

@article{shin2026hoiretarget,
  title   = {HOI-Retarget: Contact-Centric Retargeting for Human-Object Interaction},
  author  = {Shin, Jihwan and L\'opez Escoriza, Adri\`a and He, Junzhe and
             Heyrman, Matthias and Hutter, Marco},
  year    = {2026},
  note    = {Manuscript under review},
  url     = {https://shinben0327.github.io/hoi-retarget}
}

Cite the source dataset your clips come from as well — the dataset column names it, and NOTICE.md lists every reference.