PingPongPangolin/CRONOS-Benchmark

CRONOS-Benchmark dataset

CRONOS-Benchmark is a controlled, synthetic benchmark for evaluating counterfactual physical consistency in video world models. It tests whether generative video models correctly simulate three fundamental physical event types — object falling, object collision, and object occlusion — across diverse scenes, objects, and viewpoints.

Each sequence provides ground-truth RGB frames, depth maps, and segmentation masks, along with a 5-frame conditioning clip, enabling standardised evaluation of any video generation model that accepts image or video conditioning.

Dataset Details

Dataset Description

CRONOS-Benchmark dataset contains 675 sequences rendered in a controlled simulation environment. Each sequence captures one of three physical event types with foreground objects placed in a realistic indoor or outdoor scene. Ground-truth multi-modal annotations (RGB, depth, mask) are provided for all 90 frames of every sequence.

• —License: Apache 2.0

Dataset Sources

• —Repository: TBD
• —Paper: TBD

Dataset Structure

Taxonomy

Sequence counts

Directory layout

dataset/
└── {event}/                          # fall | collision | occlusion
    └── {scene}/                      # apartment | house | garden | pool | kitchen
        └── {object}/                 # Bottle | Can | TennisBall | ToyTruck | SoccerBall
            └── {appearance}/         # appearance_0 | appearance_1 | appearance_2
                └── {viewpoint}/      # close | front | side | top
                    ├── rgb/          # 90 × frame_XXXX.jpg  (1920×1080, colour)
                    ├── depth/        # 90 × frame_XXXX.jpg  (1920×1080, depth)
                    ├── mask/         # 90 × frame_XXXX.jpg  (1920×1080, binary)
                    └── movies/
                        ├── complete.mp4        # full 90-frame sequence
                        └── conditioning_5.mp4  # first 5 frames (conditioning input)

Modalities

Dataset statistics

Uses

Direct Use

CRONOS-Benchmark is designed to evaluate video generation / world models on physical event generation. Given the first frame or a conditioning clip as input, a model is asked to generate the remaining frames of the sequence.

Out-of-Scope Use

• —Real-world scene understanding (data is fully synthetic).

Dataset Creation

Source Data

Data Collection and Processing

All sequences were rendered synthetically using UnrealEngine5. All 3D assets were bought from the Unreal Engine asset store (FAB).

Personal and Sensitive Information

The dataset contains no personal or sensitive information. All content is synthetic.

Bias, Risks, and Limitations

• —Synthetic domain gap: Models trained or tuned on real video may underperform relative to their true capability due to the synthetic appearance of the scenes.
• —Limited scene and object set: Our combinatorial design limits the number of variations we can generate per visual factor. Therefore, only five scenes, five object and four views are included.

Citation

TBD