reactor-team/minecraft-vpt-tokenized

Minecraft VPT Tokenized Latents

Pre-tokenized latent episodes and aligned VPT-style actions derived from the Minecraft VPT MP4 ArrayRecord dataset. The files are msgpack-encoded ArrayRecord shards intended for dynamics and world-model training in dreamer4-jax-private.

Download

pip install -U huggingface_hub
hf download reactor-team/minecraft-vpt-tokenized \
  --repo-type dataset \
  --local-dir /path/to/tokenized_data_500M

The expected layout is:

/path/to/tokenized_data_500M/
  shard-00000.array_record
  shard-00001.array_record
  ...
  metadata/latent_stats.npz  # when included in the source data

Do not use datasets.load_dataset() for this repository: these are ArrayRecord shards rather than Parquet files or a standard Hugging Face dataset builder.

Record format

Each ArrayRecord entry is a msgpack dictionary:

{
    "latents": latent_array,    # (T, n_latents, d_bottleneck)
    "actions": actions_dict,    # action arrays aligned with T
    "source": "optional-id-or-path",
}

NumPy arrays are encoded recursively as dictionaries containing _type, data, shape, and dtype fields.

Reading records directly

pip install array-record msgpack numpy

import msgpack
import numpy as np
from array_record.python.array_record_module import ArrayRecordReader


def decode(value):
    if isinstance(value, dict):
        if value.get("_type") == "ndarray":
            return np.frombuffer(
                value["data"], dtype=value["dtype"]
            ).reshape(tuple(value["shape"]))
        return {key: decode(item) for key, item in value.items()}
    return value


reader = ArrayRecordReader(
    "/path/to/tokenized_data_500M/shard-00000.array_record"
)
packed = msgpack.unpackb(reader.read(0), raw=False)
record = {key: decode(value) for key, value in packed.items()}

print(record["latents"].shape)
print({key: value.shape for key, value in record["actions"].items()})

The same decoding logic is available as deserialize_msgpack_record in dreamer/data/serialization.py.

Using with dreamer4-jax-private

Set the dataset path and actual local shard count in configs/dataset/minecraft_vpt_latent.yaml:

name: minecraft_vpt_latent
data_type: latent
array_record_path: /path/to/tokenized_data_500M
index_max: 89

The example index_max: 89 loads shard-00000.array_record through shard-00088.array_record. Change it if the repository contains a different number of shards or if you downloaded only a contiguous subset.

The Dreamer loader uses grain.sources.ArrayRecordDataSource, deserializes the msgpack records, normalizes latents, slices temporal sequences, and batches the latents with their aligned actions.

For the tokenizer checkpoint used to produce this dataset, the example Dreamer configuration uses a bottleneck width of 16 and the following statistics:

C: 16
latent_mean: [-0.0127555020, -0.0442529507, 0.0824803188, 0.0427149609,
  0.0089577036, -0.0018820130, -0.0128933704, -0.0824453905,
  0.0111762192, -0.0944086686, -0.0582579300, -0.0497550331,
  -0.0025538108, 0.0423149280, -0.0691476464, 0.0755984560]
latent_std: [0.0848616436, 0.0906647444, 0.1006800979, 0.0940773115,
  0.1937398762, 0.0892994478, 0.0907244012, 0.1050340906,
  0.1614910215, 0.1081596166, 0.1500685960, 0.1263765246,
  0.0923914909, 0.1015426889, 0.1079730168, 0.1093038395]

Use the statistics shipped in metadata/latent_stats.npz when available, and ensure they match the tokenizer checkpoint and the latent channel width.

import numpy as np

stats = np.load("/path/to/tokenized_data_500M/metadata/latent_stats.npz")
print("latent_mean:", stats["mean"].tolist())
print("latent_std:", stats["std"].tolist())
print("num_samples:", int(stats["num_samples"]))
print("num_videos:", int(stats["num_videos"]))

Important constraints

• —Preserve contiguous shard-NNNNN.array_record filenames.
• —index_max is a shard count, not an episode count.
• —Every selected record must contain at least long_T timesteps.
• —Dynamics training requires long_T % short_T == 0 when short sequences are packed into long training examples.
• —The global batch size must be divisible by jax.process_count().
• —Actions and latents must remain temporally aligned.
• —latent_mean, latent_std, C, and the tokenizer checkpoint must agree.

Count locally available shards with:

find /path/to/tokenized_data_500M -maxdepth 1 \
  -name 'shard-*.array_record' | sort | wc -l

Intended use

This dataset is intended for research on latent dynamics modeling, action-conditioned world models, long-context sequence modeling, and Minecraft behavior prediction. It is not a drop-in replacement for raw video when pixel reconstruction is required; decoding requires the matching tokenizer model.

Users are responsible for validating the data, its provenance, and whether their intended use complies with applicable licenses, platform terms, and policies.