Zeus-Omni/D3IL-Vision

D3IL Vision + LeRobot / LingBot-VA

A local derivative of the D3IL demonstrations with two-camera vision coverage across all eight task subsets, LeRobot v2.1 trajectories, and precomputed LingBot-VA video/text features.

Upload status: See UPLOAD_STATUS.json. This repository is complete only when its status is complete. Files may appear incrementally during the initial upload.

Contents

There are 25,195 train clips and 2,878 eval clips, split by episode. The release contains 8,298 Parquet trajectories, 16,596 MP4 videos, and 56,146 per-camera latent files.

• —lerobot/<subset>/: meta/, data/, videos/, latents/, and per-subset configuration.
• —lerobot/lingbot_joint_config.json: shared action normalization and camera/channel mapping.
• —text_embeddings/: task and empty-prompt embeddings.
• —raw_vision/<subset>/part-*.tar: lossless archives of the complete original-resolution state, images, and other available rollout records. Tar members start with raw_vision/<subset>/; extract into the snapshot root.
• —raw_vision/<subset>/shards.json: archive member counts, source byte counts, and SHA256 hashes.
• —reports/, provenance/, licenses/: validation results, generation records, source snapshots, and upstream notices.
• —tools/prepare_local.py: set downloaded configuration paths to the current local directory.

Python environments, temporary pilot data, training checkpoints, account credentials, and upload-worker state are excluded.

Instructions

Each subset uses a single goal-level instruction, without prescribing a route, object order, or stacking order.

Generation and action semantics

Avoiding, Pushing, and Inserting images were generated by restoring logged robot/object states and rendering external and wrist cameras at 256×256. This is state restoration and rendering, not a fresh physical execution of each demonstration. Original terminal-state failures (one Pushing and six Inserting episodes) were excluded. Source-native vision for Aligning, Sorting, and Stacking is retained at 96×96; VAE inputs are resized to 256×256.

A source rollout with T observations yields T−1 valid transitions. Row t pairs the observation at t with the next logged desired joint positions and next observed gripper width. The terminal observation remains in the raw rollout. action and observation.state each have eight float32 channels: seven joint positions in radians and gripper total width in meters. Gripper width is a proxy target, not a recovered original gripper control command; non-grasping records without width use a documented 0.04 m open-gripper assumption. Missing Cartesian targets are not fabricated.

For LingBot-VA, the eight action channels map to indices 14–20 and 28 in its 30-dimensional action layout. Unused channels and initial synthetic action padding must be masked. This is a joint-target policy dataset; use a matching joint/gripper controller when evaluating.

Most tasks have original dt=0.035 s; Stacking has dt=0.03 s. Video sampling stride for VAE encoding is 3, and action_per_frame=12. Video latents use Wan2.2 VAE posterior mode with checkpoint normalization; text features use UMT5. The two camera keys are observation.images.bp_cam and observation.images.wrist_cam. Latents have 48 channels at 16×16 per view; text embeddings have shape 512×4096. Segments can overlap near episode ends and can have different temporal lengths. The validated native training configuration uses batch_size=1 per device.

The 1%/99% action normalization quantiles are shared across subsets and computed from training data only. The custom LingBot-VA loader used for validation honors dataset_split and mask_initial_action_padding; an unmodified upstream loader may not honor these optional fields.

Download

Download only the converted training data:

from huggingface_hub import snapshot_download
snapshot_download(
    repo_id="Zeus-Omni/D3IL-Vision",
    repo_type="dataset",
    local_dir="d3il_vision",
    allow_patterns=["lerobot/**", "text_embeddings/**", "tools/**", "README.md", "reports/**", "licenses/**"],
)

Then run:

python d3il_vision/tools/prepare_local.py --root d3il_vision

For the complete source vision release, omit allow_patterns above. Check UPLOAD_STATUS.json and each shards.json before relying on completeness. Extract each raw tar into the snapshot root, for example:

tar -xf d3il_vision/raw_vision/avoiding/part-00000.tar -C d3il_vision

Each lerobot/<subset> is an individual LeRobot v2.1 dataset. The outer lerobot directory is a collection for LingBot-VA's multi-dataset loader, not a single LeRobot dataset. Generation provenance can retain original local source paths as audit references; they are not needed for training from the converted data.

Attribution

Original demonstrations and environments: ALRhub/D3IL, commit 1d9c71850d4fc1477cee17b557c8b97d70b13071. The upstream software MIT notice is preserved in licenses/D3IL-MIT.txt.

Video/action model and data format: Robbyant/LingBot-VA. The generation environment used local lingbot-va-posttrain-robotwin encoder assets. Model weights are not included in this dataset repository. See provenance/ for the local conversion scripts and dependency versions.

This is a derived dataset release, not an official upstream D3IL or LingBot-VA release. Format/load validation does not establish closed-loop policy success or physical validity of an entire restored trajectory.