CoolFace
Datasetpublic

zimplex/genesis-hr-bench-rebuttal-full19-hrv2-eval-20260812

Genesis HR Bench — preliminary HR v2 train19 rebuttal evaluation This is an immutable, sanitized snapshot of the currently available artifacts from: hf19_eval5_act080000_full95_fast_20260812T0236Z It contains per-episode videos, state traces, result JSON, validation records, task/model manifests, policy-server logs, and Slurm logs for five policies across the 19 HR v2 tasks used for this rebuttal evaluation. It also includes both focused seed-7 skillet diagnostic pilots.… See the full description on the dataset page: https://huggingface.co/datasets/zimplex/genesis-hr-bench-rebuttal-full19-hrv2-eval-20260812.

sourceHugging Faceupdated 2mo agoView on Hugging Face
0likes165downloads
Dataset Card

Genesis HR Bench — preliminary HR v2 train19 rebuttal evaluation

This is an immutable, sanitized snapshot of the currently available artifacts from:

hf19_eval5_act080000_full95_fast_20260812T0236Z

It contains per-episode videos, state traces, result JSON, validation records, task/model manifests, policy-server logs, and Slurm logs for five policies across the 19 HR v2 tasks used for this rebuttal evaluation. It also includes both focused seed-7 skillet diagnostic pilots.

Protocol

  • —Models: ACT step 80,000; Diffusion Policy step 6,250; pi0 step 100,000; pi0.5 step 100,000; SmolVLA step 100,000
  • —Tasks: the exact 19-task hf19_20260811 profile
  • —Episodes: 10 seeds per task/model (0 through 9)
  • —Horizon: 300 policy steps for every task (howto300)
  • —Action convention: qpos-absolute
  • —Collision tracking: enabled, stride 30; safety-distance thresholds 0.05/0.10/0.20 m
  • —Main evaluation source commits: b12c5190e55ec31468e9743871dfc3869117ea50 and neutral-task compatibility retry ee31b7eb74341303a4dd5985743f90efd1d892ec

Important preliminary-status warning

Do not treat this snapshot as the final repaired aggregate. Four production trajectories—seed 7 for ACT, pi0, pi0.5, and SmolVLA on place_food_in_skillet_assist—are invalid because a falling food object caused a simulator-state collapse and downstream NaN request failure. Diffusion Policy's corresponding seed completed. Collision tracking is diagnostic and did not gate task success or cause the failure.

The first diagnostic (skillet_seed7_1a14f4f_smoke) showed that zeroing velocity plus a fail-closed guard prevented silent NaN propagation but did not make the rollout valid. The second (skillet_seed7_20ee056_smoke) captures the next candidate repair pilot. STATUS.md records the exact observed inventory and outcomes at snapshot time. A production repair retry was intentionally not folded into this preliminary snapshot.

Layout

text
production_eval/hf19_eval5_act080000_full95_fast_20260812T0236Z/
diagnostics/skillet_seed7_1a14f4f_smoke/
diagnostics/skillet_seed7_20ee056_smoke/
README.md
STATUS.md
SNAPSHOT_MANIFEST.json
SHA256SUMS

Absolute cluster paths in UTF-8 text and JSON were replaced with <GENESIS_HR_BENCH_ROOT> and <GCP_HOME>. Numeric trajectory values were otherwise left byte-for-byte unchanged, and MP4 files were copied without modification. Use SHA256SUMS to verify the snapshot payload.

Exact inference checkpoints

  • —ACT: <https://huggingface.co/zimplex/act-genesis-hr-v2-train19-qpos-abs-step080000>
  • —Diffusion Policy: <https://huggingface.co/zimplex/diffusion-genesis-hr-v2-train19-qpos-abs-step006250>
  • —pi0: <https://huggingface.co/zimplex/pi0-genesis-hr-v2-train19-qpos-abs-step100000>
  • —pi0.5: <https://huggingface.co/zimplex/pi05-genesis-hr-v2-train19-qpos-abs-step100000>
  • —SmolVLA: <https://huggingface.co/zimplex/smolvla-genesis-hr-v2-train19-qpos-abs-step100000>

The checkpoint repositories contain only their pretrained_model inference leaves, not optimizer or other training state.

zimplex/genesis-hr-bench-rebuttal-full19-hrv2-eval-20260812 · CoolFace