robium/pi05-libero-goal-task-8-evidence

Robium Pi0.5 LIBERO-Goal Task 8 evidence

This is the public evidence bundle for Robium issue #69. It records one fixed, no-retry evaluation of lerobot/pi05_libero_finetuned_v044 on LIBERO-Goal task 8, put_the_bowl_on_the_plate, using the canonical prompt “put the bowl on the plate.”

Result

• —20/20 successful episodes; the predeclared target was 16/20.
• —Fixed initial states 0–19 map to seeds 1000–1019.
• —Batch size 1, hard environment/policy reset before every episode, 300-step cap, and 10 action steps per policy inference.
• —Exactly one evaluation process and no retries.
• —First episode: 239.239 seconds. Its first compiled action took 231.214 seconds.
• —Remaining 19 episodes: 6.357 seconds mean, 6.747 seconds p95 and maximum.
• —RunPod compute: one Secure Cloud RTX PRO 4500 SE in US-KS-2 at $0.72/hour. The settled balance-window cost was $0.2252055332, including the negligible persistent-volume charge during the same window.

This is Robium's own single-task, 20-episode measurement. It is not LeRobot's published multi-suite Pi0.5 result and should not be presented as one.

Immutable inputs

The image ran PyTorch 2.10.0+cu128 on an NVIDIA RTX PRO 4500 Blackwell Server Edition (compute capability 12.0). cuda-preflight.json contains the sanitized CUDA probe.

Bundle layout

• —manifest.json: machine-readable benchmark configuration, immutable revisions, aggregate result, cost, per-episode results and video hashes.
• —episode-0.json … episode-19.json: simulator success, steps, duration, latency summary, video path and video SHA-256.
• —episode-0.mp4 … episode-19.mp4: complete 256×256 H.264 rollouts.
• —evaluation-run.json: independently checked application/image/GPU identity.
• —cuda-preflight.json and phase.json: sanitized startup and completion diagnostics.
• —SHA256SUMS: hashes for every published bundle artifact except the checksum file itself.

Reproduce

With the immutable application image and staged checkpoint:

python -m vla_pick_and_place.cli evaluate \
  --output /evidence \
  --application-commit 9aba9cba20ea18e5b98f95ccce65f242bc2a8eff \
  --image-digest sha256:1301937ab020a8120b54aefd680f40ed4191857406e9cb1e73816bfaf1270f0c \
  --gpu "NVIDIA RTX PRO 4500 Blackwell Server Edition"

The exact serialized configuration and all state/seed mappings are in manifest.json. Verify the downloaded bundle with sha256sum -c SHA256SUMS.