espejelomar/worldforge-go2-dimos-replay-world-pairs

WorldForge Go2 DimOS Replay World Pairs

This dataset is a compact, derived world-model dataset built from public `dimensionalOS/dimos` Unitree Go2 replay assets.

Companion benchmark: `go2-air-controlbench-v1` provides measured command-to-outcome trials on a real Go2 (commands, no images). This dataset provides the robot-POV image pairs (images, no commands). Together they cover the visual and control halves of the WorldForge score workflow.

New — expanded config: `world_pairs_multihorizon` has 12,849 pairs across 1 s / 2 s / 3 s / 5 s horizons, recovers the previously skipped go2_china_office replay (1,527 pairs, poses decoded from odometry), and keeps embedded current_image/future_image plus per-pair body-frame motion. The original world_pairs config below is unchanged and remains the default.

It is designed for the WorldForge score contract:

current robot-view image + candidate egomotion/action delta
-> predicted future visual latent
-> score against a goal/future latent

Contents

• —Source replay frames: 20918
• —Exported frame pairs: 2557
• —Unique exported frames: 5046
• —Splits: {"test": 383, "train": 1791, "validation": 383}
• —Source pair counts: {"go2_bigoffice": 500, "go2_china_office": 0, "go2_hongkong_office": 500, "go2_short": 203, "go2_slamabuse1": 500, "go2_slamabuse2": 500, "markers_go2": 354}

Source replay assets:

• —go2_short: data/.lfs/go2_short.db.tar.gz / 8a19846a0adf5755815fd039492c0255e0bc282e9df75a06648d7585cae8d2d2
• —markers_go2: data/.lfs/markers_go2.db.tar.gz / 5a43529f8dbc2aedcccca6ae89747235826123c2bc066e0dc8b87c2042219dae
• —go2_bigoffice: data/.lfs/go2_bigoffice.db.tar.gz / e66f5472e72f370446d8dcd802f70f3c3c07e4e083c5d6a394873877dec4c88d
• —go2_hongkong_office: data/.lfs/go2_hongkong_office.db.tar.gz / d1bb7de9a090b4053ba1ee4f36d776e439d970cba08ebb489f9311f26946f56c
• —go2_slamabuse1: data/.lfs/go2_slamabuse1.db.tar.gz / a85feac43debdebf344c567483ab7d1bec12c3cf9e4df26034260a24e225f219
• —go2_slamabuse2: data/.lfs/go2_slamabuse2.db.tar.gz / 7d9a13596cf3d9a50e437fa89e8a3d68d843587116681564b4de7422b53c54dd
• —go2_china_office: data/.lfs/go2_china_office.db.tar.gz / 834539871fd325b15f3079a3490b278c54e78d0d40bfa1342dbdc983f6a3ee02

Each row includes:

• —current_image and future_image as decoded Image features (in the default world_pairs config)
• —timestamps, poses, and egomotion_delta (robot-body-frame motion over the pair)
• —z_drift_m and pose_quality for filtering noisy SLAM poses (see below)
• —from_slamabuse_source to optionally drop the two slamabuse stress replays
• —pair_preview_path side-by-side preview for the Hugging Face image viewer
• —the explicit world-model score_contract

The repository also includes imagefolder/train, imagefolder/validation, and imagefolder/test directories. Each split has pair-preview JPEGs plus a metadata.jsonl file with the same labels, so it can be loaded with the standard Hugging Face imagefolder builder.

How To Load

The default world_pairs config stores current_image and future_image as real decoded Image features, so loading the trainable pairs is one line:

from datasets import load_dataset

ds = load_dataset("espejelomar/worldforge-go2-dimos-replay-world-pairs", split="train")
row = ds[0]
row["current_image"]    # PIL.Image, the robot view now
row["future_image"]     # PIL.Image, ~3 s later
row["egomotion_delta"]  # dx_body_m forward, dy_body_m left, dyaw_rad
row["z_drift_m"], row["pose_quality"]

Passing "world_pairs" explicitly is equivalent, since it is the default config.

Filtering noisy poses

The robot is a ground quadruped, so vertical motion over a 3 s pair should be near zero. z_drift_m = future_pose.z_m - current_pose.z_m is therefore a direct proxy for SLAM vertical drift, and pose_quality buckets it:

clean = ds.filter(lambda r: r["pose_quality"] == "good")
# stricter: also drop the two slamabuse stress replays
strict = ds.filter(lambda r: r["pose_quality"] == "good" and not r["from_slamabuse_source"])

Only ~2% of pairs drift more than 5 cm in z, so the slamabuse source label over-counts bad poses. Prefer pose_quality for per-pair filtering and use from_slamabuse_source only when you want to be conservative.

Raw frames on disk

To work from the original per-split JSONL and full file tree (for example to read frames straight from images/frames/), download the repository snapshot and open the frame paths directly:

import json
import os

from huggingface_hub import snapshot_download
from PIL import Image

root = snapshot_download(
    "espejelomar/worldforge-go2-dimos-replay-world-pairs",
    repo_type="dataset",
)


def load_split(split):  # split in {"train", "validation", "test"}
    with open(os.path.join(root, "data", f"{split}.jsonl")) as f:
        for line in f:
            row = json.loads(line)
            current = Image.open(os.path.join(root, row["current_image"]))
            future = Image.open(os.path.join(root, row["future_image"]))
            delta = row["egomotion_delta"]  # dx_body_m, dy_body_m, dyaw_rad, ...
            yield current, future, delta


for current, future, delta in load_split("train"):
    # current robot view + candidate egomotion delta -> predict / score future view
    ...

Frames are 480x270 JPEGs under images/frames/; egomotion_delta is expressed in the robot body frame (dx_body_m forward, dy_body_m left, dyaw_rad).

Expanded config: world_pairs_multihorizon

world_pairs_multihorizon is a larger, denser rebuild from the same source replays. It is a separate, non-default config, so existing world_pairs users are not affected.

What is different from the default world_pairs:

Counts:

• —Pairs by source: go2_short 654, markers_go2 1137, go2_bigoffice 2400, go2_hongkong_office 2400, go2_slamabuse1 2331, go2_slamabuse2 2400, go2_china_office 1527.
• —Pairs by horizon: 1 s 3241, 2 s 3227, 3 s 3209, 5 s 3172.
• —Splits (temporal, per trajectory): train 9136, validation 1953, test 1760.
• —pose_quality: good 12687 (98.7%), suspect 101 (0.8%), bad 61 (0.5%).

Each row is flattened for direct columnar use: current_image, future_image (Image), horizon_bucket_s (nominal 1/2/3/5) and horizon_s (actual elapsed seconds), current_x_m/.../current_yaw_rad, future_x_m/.../future_yaw_rad, body-frame dx_body_m/dy_body_m/dyaw_rad, distance_m, z_drift_m, pose_quality, and pose_source.

from datasets import load_dataset

ds = load_dataset(
    "espejelomar/worldforge-go2-dimos-replay-world-pairs",
    "world_pairs_multihorizon",
    split="train",
)
row = ds[0]
row["current_image"], row["future_image"]            # PIL.Image now and horizon later
row["horizon_bucket_s"]                               # 1.0 / 2.0 / 3.0 / 5.0
row["dx_body_m"], row["dy_body_m"], row["dyaw_rad"]   # body-frame egomotion over the pair

three_sec = ds.filter(lambda r: r["horizon_bucket_s"] == 3.0)
clean = ds.filter(lambda r: r["pose_quality"] == "good")

Pose provenance for this config

• —For the six replays with valid pose columns, poses come straight from the color_image pose columns (pose_source = "column").
• —go2_china_office has null pose columns in the published .db, so its poses are decoded from the LCM odometry blob (odom_blob, the trailing 7 doubles x, y, z, qx, qy, qz, qw) and matched to the nearest image timestamp (pose_source = "odom_blob_nearest_ts"). This is why go2_china_office is now usable instead of skipped. Recovered trajectories span several meters and scale correctly with horizon, but the per-frame timestamp match adds a small extra alignment error relative to the column-pose sources; filter on pose_source if you need a homogeneous subset. The exact decode parameters and validation evidence (byte order, decoded tuple order, timestamp-match tolerance, trajectory span, and distance-by-horizon) are recorded in `metadata/china_office_pose_decode_validation.json`.

Provenance

The source material comes from dimensionalOS/dimos, whose checked-in LICENSE file is Apache License 2.0. This repository currently reports license metadata as Other on GitHub, so users should verify the source license text directly.

Intended Use

This dataset is intended for:

• —small latent-dynamics demos,
• —action-conditioned future prediction experiments,
• —WorldForge score-provider prototyping,
• —educational robotics evidence-trace examples.

Limitations

• —This is not a broad robot foundation dataset.
• —It is a small replay-derived dataset.
• —The action labels are derived from pose deltas between frames, not raw joystick commands.
• —It is not suitable for safety validation or direct robot control.
• —Indoor replay imagery may contain real-world office context.

Citation / Attribution

If you use this dataset, attribute both:

• —DimensionalOS / DimOS as the source of the public replay data.
• —WorldForge Go2 Trace Judge as the derived dataset/scoring package.