shuaishuaicdp/robodojo-hybrid-episodes

RoboDojo Hybrid GPT and Pi 0.5 Compact Episodes

This release retains all 258 started attempts: 216 canonical complete cases (54 task variants, four repetitions each), eight complete pilot episodes, and 34 superseded or interrupted attempts. Canonical results include 57 successes and 159 failures. Failed complete episodes are retained. Seven attempts have no aligned recording; their explicit manifests and available model behavior remain present. indexes/episodes.jsonl lists every attempt.

Layout and loading

The root is an official LeRobot v3.0 dataset with 216 episodes. subsets/pilot contains eight episodes. subsets/attempts contains the safely recorded prefixes and seven unconverted manifests; these attempts are ineligible for training. Validation uses official LeRobot 0.4.1 commit a5b29d430105f5235eb05bbf2db5a0d747a869d6 with PyAV.

from pathlib import Path
import sys
from huggingface_hub import snapshot_download
from lerobot.datasets.lerobot_dataset import LeRobotDataset

root = Path(snapshot_download("shuaishuaicdp/robodojo-hybrid-episodes", repo_type="dataset"))
sys.path.insert(0, str(root / "code"))
from compact_training import ValidActionDataset
ds = LeRobotDataset("shuaishuaicdp/robodojo-hybrid-episodes", root=root, video_backend="pyav")
train = ValidActionDataset(ds, horizon=16)
sample = train[0]
# For the pilot or interrupted prefixes, use root / "subsets/pilot" or
# root / "subsets/attempts" as the local dataset root.

Action alignment and terminal masks

Parquet row k stores observation boundary k and the outgoing action from recorded step k+1. Every episode includes its final observation: its action is 14 NaNs with action.valid=false. training.episode_eligible excludes incomplete and invalid attempts. Every target in an action window must be valid, eligible, consecutive, and inside one episode. Use `compact_training.py`; the historical converter's embedded training wrapper is not the public helper. Complete failures remain eligible by default; success-only filtering is optional. Action statistics exclude invalid terminal actions. Stored JSON action/state precision is float64; original non-RGB NPZ arrays retain their dtype and bytes. The vanilla upstream tensor transform can narrow float64. The supplied training wrapper constructs exact float64 action targets from Arrow. observation.native_complete is corrected from native terminated OR truncated and independently audited against preserved step JSON before release.

RGB quality and exact retained data

All three native 640 x 480 camera streams retain every frame at 25 Hz. Each episode starts with H.264 CRF18, veryfast, YUV420p. The explicit quality ladder retries CRF16, CRF14, then CRF12 if any camera falls below 35 dB full-stream PSNR; the final fallback is byte-exact RGB using libx264rgb CRF0. Receipts record each attempted setting and the accepted per-episode codec/CRF. Complete decoding and the same quality threshold apply throughout. H.264/YUV episodes have lossy RGB: their original recording RGB pixels cannot be recovered after deletion of dense source arrays. RGB-lossless fallback episodes retain exact decoded RGB; NPZ ZIP container bytes are still not preserved. Numeric state/action values, all non-RGB arrays, recording JSON, actual GPT input images, visible model text and behavior remain exact. Per-camera errors, original hashes, encoder settings and source mappings are retained in episode receipts.

Full visible model behavior and SFT conversations

behavior/<cohort>/<task>/<episode_key>.tar.zst stores content-addressed objects and reversible recipes for every selected behavior file: prompts, tool calls and results, Pi proposals, approval/edit/execution records, input images, raw captures, and controller/online diagnostics. Repeated image Base64 and compressed NPZ payloads share objects. Reconstruction was SHA-256 checked for every original selected file, including byte-identical original text and raw transport bytes. This archive is a CAS container; extracting its tar alone does not restore the original member files. Use the included restorer:

import json
from compact_behavior import restore_archive
episode = json.loads((root / "indexes/episodes.jsonl").read_text().splitlines()[0])
meta = json.loads((root / episode["metadata_file"]).read_text())
receipt = json.loads((root / meta["behavior"]["receipt"]["path"]).read_text())
archive = next(a for a in receipt["artifacts"] if a["kind"] == "behavior_cas")
restore_archive(root / archive["path"], archive["sha256"], Path("restored-episode"))

Install the zstd executable for restoration. The included Python restorer uses the standard library plus that executable. indexes/behavior/...jsonl.gz maps each original absolute source path to a portable member, bytes, SHA-256 and source stat identity. Absolute paths are provenance only, not required destinations. conversations/...messages.jsonl.gz is the byte-exact original exported SFT-style conversation, compressed for convenient inspection. After restoration, resolve its assets/... image references relative to conversation/export/. Image assets remain exact in the behavior archive. Source flags such as sft_upload_ready=false are retained; this review format does not certify training qualification. Proposal arrays and actually executed actions remain distinct.

Original source text, including Chinese/CJK strings, is preserved verbatim. New documentation is English. All captured visible inputs, outputs and tool use are retained where recorded. Provider-hidden reasoning, unavailable HTTP internals and unexposed private context were not captured and are not reconstructed. Native simulator diagnostics do not imply that GPT or Pi observed those fields.

Videos and provenance

review/raw/<cohort>/<task>/<episode_key>/ preserves all 251 original MP4s byte-for-byte. 16 already corrupt interrupted MP4s are marked diagnostic-only; every other raw MP4 passed fresh header and full decode checks. The annotated example shows three camera views, a Pi/GPT colored timeline, goals and fixed pages of GPT tool use. Its viewing time is stretched for readability; raw video follows control time. GPT thinking pauses only its own episode's physics clock.

Model settings remain gpt-6-astra / openai / high, with the existing Codex backend and RoboDojo-finetuned Pi0.5 OpenPI/JAX checkpoint step59999, config pi05_base_aloha_full_sim_arx-x5_seed_0. The checkpoint digest is d15fb8bd1d29cb30b69f01b71c66596cb0293c1a8a94111b343c1580dd3e3e5b. The Pi action horizon is50, action dimension14, control frequency25Hz; full effective settings and transforms are in 48 shared run provenance files. The campaign adds200 controls to each native horizon; the pilot retains its original horizon. The shared service uses the original B1 model entry point; these records do not demonstrate vectorized B>1 inference or same-episode realtime chunking.

See source audit, all episodes, canonical cases, and release manifest. The manifest reports actual staged bytes, hashes and local validation coverage. Local validation is separate from HF publication: remote verification uses a pinned commit, file sizes, LFS SHA-256 or Git blob hashes, and downloaded loader/ restore checks. Cleanup requires separate authorized remote-verification receipts. This dataset contains episode data and model provenance, not checkpoint weights.

Behavior-v3 deployment

See credential policy and exact deployment provenance.