datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
financial-world-model
Twelve Data World Model Dataset
A multi-modal financial time-series dataset built from Twelve Data
market data. Each timeframe is published in three parallel views:
bars_* — OHLCV bars enriched with causal technical indicators and macro
context, in Parquet.
text_* — instruction-tuning prompts/labels derived from the bars, in
JSONL.
trajectories_* — fixed-length rolling windows of state vectors plus
next-state pairs, suitable for world-model / sequence-model training, in… See the full description on the dataset page: https://huggingface.co/datasets/twelvedata/financial-world-model.world-model-physics
Rapidata Physics Benchmark
Built by Rapidata.
Do video and world models understand physics? We gave 25 video- and world models the same
real-world starting frame and scene description from Physics-IQ and asked
them to predict what happens next. ~283,000 human votes, collected with the
Rapidata Python SDK, decided which continuation is more realistic — with the
real recording competing as a hidden 26th participant.
Each row is a head-to-head matchup between two participants on… See the full description on the dataset page: https://huggingface.co/datasets/Rapidata/world-model-physics.repro-causal-jepa-learning-world-models-through-object-level-latent-masking-traces
Agent traces
Agent sessions published from a Trackio Logbook.
webarena-world-model-cotSee https://github.com/thuml/RLVR-World for examples for using this dataset.
Citation
@article{wu2025rlvr,
title={RLVR-World: Training World Models with Reinforcement Learning},
author={Jialong Wu and Shaofeng Yin and Ningya Feng and Mingsheng Long},
journal={arXiv preprint arXiv:2505.13934},
year={2025},
}
world_model_real_rollout_genworld_model_for_wa_desc_with_tao_dataset
Dataset Card for "world_model_for_wa_desc_with_tao_dataset"
More Information needed
lerobot_pick_and_place_dataset_world_modelThis dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v2.1",
"robot_type": "so100",
"total_episodes": 30,
"total_frames": 13572,
"total_tasks": 1,
"total_videos": 30,
"total_chunks": 1,
"chunks_size": 1000,
"fps": 30,
"splits": {
"train": "0:30"},
"data_path": "data/chunk-{episode_chunk:03d}/episode_{episode_index:06d}.parquet",
"video_path":… See the full description on the dataset page: https://huggingface.co/datasets/Clementppr/lerobot_pick_and_place_dataset_world_model.so101-task2-720p-whole-arm-v3-cleanedThis dataset was created using LeRobot.
Dataset Description
Recovered and cleaned SO-101 task 2 dataset. Bad final source episodes 95 and 96 were removed. V3 data, episode metadata, and video shard indices are compact and contiguous.
License: apache-2.0
Dataset Structure
meta/info.json:
{
"codebase_version": "v3.0",
"robot_type": "so_follower",
"total_episodes": 95,
"total_frames": 73988,
"total_tasks": 1,
"chunks_size": 1000… See the full description on the dataset page: https://huggingface.co/datasets/rl26-world-models/so101-task2-720p-whole-arm-v3-cleaned.world-models-eval
DreamGrasp: Processed LIBERO Manipulation Demonstrations
Does a robot policy's evaluation still mean something if it never touched a real simulator, only a world model's imagination of one?
This dataset is the shared training data behind that question, a single, ready-to-train release built from LIBERO's manipulation demonstrations (libero_spatial, libero_object, libero_goal). It provides:
Fixed, versioned train / validation / test / held-out splits, so every result trained on… See the full description on the dataset page: https://huggingface.co/datasets/ZaidGhazal/world-models-eval.marl-world-model-lerobotworld_model_eval_logsso101-task2-720p-whole-arm-v4-fresh-trimmedThis dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v3.0",
"robot_type": "so_follower",
"total_episodes": 50,
"total_frames": 38664,
"total_tasks": 1,
"chunks_size": 1000,
"data_files_size_in_mb": 100,
"video_files_size_in_mb": 200,
"fps": 30,
"splits": {
"train": "0:50"
},
"data_path": "data/chunk-{chunk_index:03d}/file-{file_index:03d}.parquet",
"video_path":… See the full description on the dataset page: https://huggingface.co/datasets/rl26-world-models/so101-task2-720p-whole-arm-v4-fresh-trimmed.so101-task2-720p-whole-arm-v4-freshThis dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v3.0",
"robot_type": "so_follower",
"total_episodes": 50,
"total_frames": 38664,
"total_tasks": 1,
"chunks_size": 1000,
"data_files_size_in_mb": 100,
"video_files_size_in_mb": 200,
"fps": 30,
"splits": {
"train": "0:50"
},
"data_path": "data/chunk-{chunk_index:03d}/file-{file_index:03d}.parquet",
"video_path":… See the full description on the dataset page: https://huggingface.co/datasets/rl26-world-models/so101-task2-720p-whole-arm-v4-fresh.so101-task2-720p-whole-arm-v3-cleaned-trimmedThis dataset was created using LeRobot.
Dataset Description
Recovered and cleaned SO-101 task 2 dataset. Bad final source episodes 95 and 96 were removed. V3 data, episode metadata, and video shard indices are compact and contiguous.
License: apache-2.0
Dataset Structure
meta/info.json:
{
"codebase_version": "v3.0",
"robot_type": "so_follower",
"total_episodes": 95,
"total_frames": 73988,
"total_tasks": 1,
"chunks_size": 1000… See the full description on the dataset page: https://huggingface.co/datasets/rl26-world-models/so101-task2-720p-whole-arm-v3-cleaned-trimmed.world_model_for_wa_acctree_dataset_14K
Dataset Card for "world_model_for_wa_acctree_dataset_14K"
More Information needed
inference-viz-1This dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v3.0",
"robot_type": "so101_follower",
"total_episodes": 1,
"total_frames": 60,
"total_tasks": 1,
"chunks_size": 1000,
"data_files_size_in_mb": 100,
"video_files_size_in_mb": 200,
"fps": 30,
"splits": {
"train": "0:1"
},
"data_path": "data/chunk-{chunk_index:03d}/file-{file_index:03d}.parquet",
"video_path":… See the full description on the dataset page: https://huggingface.co/datasets/rl26-world-models/inference-viz-1.sokoban_easy_v8_noncot_chunk_k10_world_model_20260622_perseg
sokoban_easy_v8_noncot_chunk_k10_world_model_20260622_perseg
Sokoban action-conditioned visual world-model SFT data (non-CoT baseline) for the BAGEL-7B-MoT
VLM-Gym feedback-interval study.
Format: gzipped JSONL shards under training/, one packed row = one episode. Frames are
base64 JPEG (q95). Per-segment CoT layout: <think> per-step imagined frame (MSE target) </think>
then the committed action chunk; between chunks a loss-0 "Action executed." + real frame
(GT re-grounding).… See the full description on the dataset page: https://huggingface.co/datasets/ultrastar111/sokoban_easy_v8_noncot_chunk_k10_world_model_20260622_perseg.SO101-world-model-5fpsThis dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v3.0",
"robot_type": "so_follower",
"total_episodes": 1,
"total_frames": 107,
"total_tasks": 1,
"chunks_size": 1000,
"data_files_size_in_mb": 100,
"video_files_size_in_mb": 200,
"fps": 5,
"splits": {
"train": "0:1"
},
"data_path": "data/chunk-{chunk_index:03d}/file-{file_index:03d}.parquet",
"video_path":… See the full description on the dataset page: https://huggingface.co/datasets/julie-trrsr/SO101-world-model-5fps.so101-task1-720p-whole-arm-trimmed-subsampled-10fpsThis dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v3.0","robot_type": "so_follower",
"total_episodes": 201,
"total_frames": 62169,
"total_tasks": 1,
"chunks_size": 1000,
"data_files_size_in_mb": 100,"video_files_size_in_mb": 200,
"fps": 30,
"splits": {
"train": "0:201"
},
"data_path": "data/chunk-{chunk_index:03d}/file-{file_index:03d}.parquet",
"video_path":… See the full description on the dataset page: https://huggingface.co/datasets/rl26-world-models/so101-task1-720p-whole-arm-trimmed-subsampled-10fps.so101-task1-720p-whole-arm-trimmedThis dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v3.0",
"robot_type": "so_follower",
"total_episodes": 201,
"total_frames": 62169,
"total_tasks": 1,
"chunks_size": 1000,
"data_files_size_in_mb": 100,
"video_files_size_in_mb": 200,
"fps": 30,
"splits": {
"train": "0:201"
},
"data_path": "data/chunk-{chunk_index:03d}/file-{file_index:03d}.parquet",
"video_path":… See the full description on the dataset page: https://huggingface.co/datasets/rl26-world-models/so101-task1-720p-whole-arm-trimmed.sokoban_easy_v8_noncot_chunk_k5_world_model_20260622_perseg
sokoban_easy_v8_noncot_chunk_k5_world_model_20260622_perseg
Sokoban action-conditioned visual world-model SFT data (non-CoT baseline) for the BAGEL-7B-MoT
VLM-Gym feedback-interval study.
Format: gzipped JSONL shards under training/, one packed row = one episode. Frames are
base64 JPEG (q95). Per-segment CoT layout: <think> per-step imagined frame (MSE target) </think>
then the committed action chunk; between chunks a loss-0 "Action executed." + real frame
(GT re-grounding).… See the full description on the dataset page: https://huggingface.co/datasets/ultrastar111/sokoban_easy_v8_noncot_chunk_k5_world_model_20260622_perseg.so101-task2-720p-whole-arm-v3-cleaned-trimmed-subsampled-10fpsso101-task2-720p-whole-arm-cube-trimmedsokoban_easy_v8_cot_chunk_kinf_world_model_20260707_perseg
sokoban_easy_v8_cot_chunk_kinf_world_model_20260707_perseg
Sokoban action-conditioned visual world-model SFT data (CoT self-rollout) for the BAGEL-7B-MoT
VLM-Gym feedback-interval study.
Format: gzipped JSONL shards under training/, one packed row = one episode. Frames are
base64 JPEG (q95). Per-segment CoT layout: <think> per-step imagined frame (MSE target) </think>
then the committed action chunk; between chunks a loss-0 "Action executed." + real frame
(GT re-grounding). See… See the full description on the dataset page: https://huggingface.co/datasets/ultrastar111/sokoban_easy_v8_cot_chunk_kinf_world_model_20260707_perseg.sokoban_easy_v8_noncot_chunk_k3_world_model_20260622_perseg
sokoban_easy_v8_noncot_chunk_k3_world_model_20260622_perseg
Sokoban action-conditioned visual world-model SFT data (non-CoT baseline) for the BAGEL-7B-MoT
VLM-Gym feedback-interval study.
Format: gzipped JSONL shards under training/, one packed row = one episode. Frames are
base64 JPEG (q95). Per-segment CoT layout: <think> per-step imagined frame (MSE target) </think>
then the committed action chunk; between chunks a loss-0 "Action executed." + real frame
(GT re-grounding).… See the full description on the dataset page: https://huggingface.co/datasets/ultrastar111/sokoban_easy_v8_noncot_chunk_k3_world_model_20260622_perseg.so101-task1-720p-whole-arm-cube-trimmedworld_model_for_wa_train_dataset
Dataset Card for "world_model_for_wa_train_dataset"
More Information needed
world_model_for_wa_tao_datasetworld_model_for_wa_qa_datasetworld_model_for_wa_desc_with_tao_dataset_with_transition_count