reactor-team/minecraft-vpt-mp4

Minecraft VPT MP4 ArrayRecords

Minecraft gameplay clips and their aligned VPT-style actions, packaged as sharded ArrayRecord files. This is the raw-video dataset used by the Minecraft data path in dreamer4-jax-private.

Download

Install the Hugging Face CLI and download the repository to a local directory:

pip install -U huggingface_hub
hf download reactor-team/minecraft-vpt-mp4 \
  --repo-type dataset \
  --local-dir /path/to/mp4-arrayrecords

The downloaded directory contains contiguous shards:

/path/to/mp4-arrayrecords/
  shard-00000.array_record
  shard-00001.array_record
  ...

The complete dataset is approximately 1 TiB. To download only selected shards, use --include, for example:

hf download reactor-team/minecraft-vpt-mp4 \
  --repo-type dataset \
  --include 'shard-0000*.array_record' \
  --local-dir /path/to/mp4-arrayrecords

Record format

Each ArrayRecord entry is a pickled Python dictionary with this structure:

{
    "video": mp4_bytes,               # MP4-encoded clip as bytes
    "video_shape": (T, H, W, C),      # used for length filtering
    "actions": [action_0, ..., action_T_minus_1],
    "source": "optional-id-or-path",
}

actions contains VPT-style action dictionaries aligned with the video frames. The source field is optional.

Do not use datasets.load_dataset() for this repository: the files are ArrayRecord shards rather than Parquet or a standard Hugging Face dataset builder.

Reading records directly

Install ArrayRecord:

pip install array-record

Read and decode a record:

import io
import pickle

import decord
from array_record.python.array_record_module import ArrayRecordReader

reader = ArrayRecordReader(
    "/path/to/mp4-arrayrecords/shard-00000.array_record"
)
record = pickle.loads(reader.read(0))

video = decord.VideoReader(io.BytesIO(record["video"]), ctx=decord.cpu(0))
frames = video.get_batch(range(len(video))).asnumpy()  # (T, H, W, C)
actions = record["actions"]

assert len(frames) == len(actions)
print(record["video_shape"], frames.shape)

Using with dreamer4-jax-private

Set the raw dataset path and shard count in configs/dataset/minecraft_vpt.yaml:

name: minecraft_vpt
data_type: video
array_record_path: /path/to/mp4-arrayrecords
index_max: 1649

index_max is the number of contiguous shards to load, not the number of videos. The path builder expands the example above to shard-00000.array_record through shard-01648.array_record.

The Dreamer loader uses grain.sources.ArrayRecordDataSource, unpickles each record, decodes the MP4 bytes, parses the aligned VPT actions, and applies the configured temporal sampling and normalization.

For offline tokenization, use scripts/tokenize_minecraft_dataset.py. It reads full episodes and writes msgpack-encoded latent ArrayRecords. Videos grouped in the same tokenization batch must have compatible shapes.

Important constraints

• —Shards must retain their contiguous shard-NNNNN.array_record names.
• —Configure index_max to match the number of shards you downloaded.
• —Each selected record must contain at least the configured number of frames.
• —The action sequence must remain aligned with the video frames.
• —Pixel normalization statistics should be recomputed if the selected data distribution changes.
• —For the provided 360x640 setup, the Dreamer configuration pads the height to 368 so both spatial dimensions are divisible by a patch size of 16.

Count locally available shards with:

find /path/to/mp4-arrayrecords -maxdepth 1 \
  -name 'shard-*.array_record' | sort | wc -l

Intended use

This dataset is intended for research on video representation learning, Minecraft behavior modeling, action-conditioned video prediction, offline tokenization, and world-model training.

Users are responsible for validating the data, its provenance, and whether their intended use complies with applicable licenses, platform terms, and policies.