sjmathy/interactive-video-reasoning
Dataset release Videos, trajectories, manifests, and evaluation records are hosted at: https://huggingface.co/datasets/sjmathy/interactive-video-reasoning Expected layout: data/ balance_rod/{task_bank,probe_bank}/ pushbar_finishline/{task_bank,probe_bank}/ slide_block/{task,probe}/ notched_launcher/shards/{task,probe}/ Domain Task bank Probe bank Held-out rule Balance-Rod 500 episodes 500 episodes pose groups 80-99 Push-Bar 500 episodes 500 episodes pose… See the full description on the dataset page: https://huggingface.co/datasets/sjmathy/interactive-video-reasoning.
0168
1# Simulator setup contract2 3## Isaac domains: Sliding, Rope, and Bankshot4 5The evaluated jobs used the NVIDIA Isaac Lab6`3.0.0-beta2-post1` runtime image. The cluster-local SIF is not included: it7contains NVIDIA Isaac Sim and must be obtained under NVIDIA's license.8 91. Install the matching NVIDIA Isaac Sim/Isaac Lab release from NVIDIA.102. Extract `task2_simulator_and_evaluation_sources_20260907.tar.gz` at the11 repository root.123. Set `PYTHONPATH` to the repository root, `reference/vilp`, and13 `pusht_vilp` as shown in the supplied `.sbatch` files.144. Replace cluster-local absolute paths in the supplied launch scripts with15 the downloaded Hugging Face artifact paths.165. Verify every artifact against `ARTIFACT_MANIFEST.json` before execution.17 18The Bankshot evaluator additionally reads19`evaluation_assets/bankshot/context_bundle.pt`, `probe_features.pt`, and the20`probes/` directory. The Sliding evaluator reads21`evaluation_assets/sliding/protocol_inputs.pt`. The Rope evaluator's small22legacy bootstrap/calibration bindings are retained under23`evaluation_assets/rope/` because the exact evaluation entry point validates24their hashes before loading the unified updater.25 26The source archive contains our Isaac task contracts, collectors, rendering27settings, K=4 materializers, WAM/VQ loaders, persistent updater integration,28closed-loop evaluators, and result aggregators. It does not contain Isaac Sim.29 30## Pouring domain: FluidLab31 32The pouring environment is reconstructed against upstream FluidLab commit:33 34```text35repository: https://github.com/zhouxian/FluidLab.git36commit: 7295eb6ed7ee92b2be471fd30b432238577f28ce37python: 3.10.1938taichi: 1.7.439```40 41After checking out that commit, apply:42 43```bash44git apply fluidlab_task2_patch_7295eb6_20260907.patch45```46 47The patch adds the hidden-physics thin-receiver benchmark and the exact local48changes used for physical collection/rendering. The task source archive49contains the Pouring data preparation, VQ/WAM training wrappers, Stage-250integration, outcome-bank evaluator, and aggregation code.51 52The released Pouring four-arm run uses the production-scale Stage-1 WAM bound53to SHA-256 `997a09823d6cfab0b519416773f04524810ff1a0d0807b9d27f1fa0928fbc256`.54Its exact evaluator also consumes `evaluation_assets/pouring/eval_probe_vjepa/`55and the 75-entry `evaluation_assets/pouring/outcomes_support_v2/` bank.56