petersonco/worldsimprobe

WorldSimProbe Public Evaluation Inputs

This is the public input release for WorldSimProbe, an action-conditioned video generation benchmark for embodied world models. It contains separate, self-contained evaluation packages for RoboTwin, LIBERO, and ManiSkill.

Ground-truth videos, evaluator annotations, simulator outcomes, private ID maps, and task-specific hidden labels are intentionally excluded. The official evaluator holds those materials separately.

Project page: https://evophys.com/WorldSimProbe/

Choose a backend

Each backend directory includes its own:

• —README.md with the complete generation contract;
• —public_samples.jsonl manifest;
• —strict public_samples.schema.json;
• —release audit in public_samples.validation.json;
• —initial-context PNGs under assets/context/; and
• —action trajectories under actions/.

Use the backend README as the authoritative data contract. Do not mix action spaces, cameras, resolutions, or action rates across backends.

Common public row contract

All manifests use the same compact public shape:

{
  "schema_version": "worldsimprobe.public_sample.v1",
  "sample_id": "wsp_<opaque id>",
  "task_id": "task1|task2|task3|task4|task5",
  "model_input": {
    "instruction": "...",
    "camera": "...",
    "context_frames": [{"path": "...", "role": "initial", "frame_index": 0, "timestamp_sec": 0.0}],
    "eval_timestamps_sec": [0.0, 0.1, 0.2],
    "timing": {"action_hz": 10.0, "video_fps": 10.0, "timebase": "seconds_from_initial_frame"},
    "action_trajectory": {"path": "...", "format": "npy", "dtype": "float32", "shape": [3, 14]}
  }
}

The example is illustrative. Backend schemas define the exact camera, action dimension, timing, paths, and required fields. Task 1 additionally supplies the three action roles original, small, and large.

Generation and submission

For every public row:

1. 1.Load the instruction, single initial-context image, and required action trajectory.
2. 2.Generate the complete horizon at the row's requested output timestamps.
3. 3.For Task 1, submit original, small, and large videos. For Tasks 2-5, submit one candidate video.
4. 4.Produce one generation per required sample/role.
5. 5.Preserve the opaque sample_id exactly in the submission manifest.

Use MP4/H.264 output for maximum compatibility. Validate submissions with the public WorldSimProbe toolkit before evaluation.

Loading from Hugging Face

The repository exposes three dataset configurations:

from datasets import load_dataset

robotwin = load_dataset(
    "petersonco/worldsimprobe",
    "robotwin",
    split="test",
)
libero = load_dataset(
    "petersonco/worldsimprobe",
    "libero",
    split="test",
)
maniskill = load_dataset(
    "petersonco/worldsimprobe",
    "maniskill",
    split="test",
)

For video generation, download the full selected backend directory so the relative PNG and NPY paths resolve locally.

Integrity and privacy

All packages use opaque identifiers and strict schemas that reject unknown fields. Their validation reports verify manifest hashes, action shapes and dtypes, image decoding, asset closure, and the absence of internal paths and private evaluator fields.

No raw episode IDs, reference videos, reference HDF5 files, primitive labels, simulator success values, or provenance paths are included.

License and attribution

WorldSimProbe release materials are provided under the MIT License. Each backend directory documents applicable upstream attribution and third-party notices. See `LICENSE` and the backend READMEs before redistribution.