TESS-Computer/minecraft-vla-stage1
Minecraft VLA Stage 1: Action Pretraining Data Vision-Language-Action training data for Minecraft, processed from OpenAI's VPT contractor dataset. Dataset Description This dataset contains frame-action pairs from Minecraft gameplay, designed for training VLA models following the Lumine methodology. Source Original: OpenAI VPT Contractor Data (7.x subset) Videos: 17,886 videos (330 hours of early-game gameplay) Task: "Play Minecraft" with focus on… See the full description on the dataset page: https://huggingface.co/datasets/TESS-Computer/minecraft-vla-stage1.
31.6k
1---2license: mit3task_categories:4 - robotics5 - video-classification6tags:7 - minecraft8 - vla9 - vision-language-action10 - gaming11 - behavioral-cloning12size_categories:13 - 1M<n<10M14---15 16# Minecraft VLA Stage 1: Action Pretraining Data17 18Vision-Language-Action training data for Minecraft, processed from OpenAI's VPT contractor dataset.19 20## Dataset Description21 22This dataset contains frame-action pairs from Minecraft gameplay, designed for training VLA models following the [Lumine](https://www.lumine-ai.org/) methodology.23 24### Source25- **Original**: [OpenAI VPT Contractor Data](https://github.com/openai/Video-Pre-Training) (7.x subset)26- **Videos**: ~17,886 videos (~330 hours of early-game gameplay)27- **Task**: "Play Minecraft" with focus on first 30 minutes of new worlds28 29### Format30 31Each sample contains:32| Field | Type | Description |33|-------|------|-------------|34| `image` | bytes | 640x360 JPEG frame |35| `video_id` | string | Source video identifier |36| `frame_idx` | int | Frame number at 5Hz |37| `action` | string | Lumine-format action string |38 39### Action Format40 41```42<|action_start|> mouse_x mouse_y scroll ; K1 ; K2 ; K3 ; K4 <|action_end|>43```44 45- `mouse_x`, `mouse_y`: Mouse delta (-1000 to 1000)46- `scroll`: Hotbar scroll (always 0 - VPT uses number keys)47- `K1` to `K4`: Key combinations per 50ms chunk48 49**Example:**50```51<|action_start|> 45 -12 0 ; W ; W Space ; W LMB ; W LMB <|action_end|>52```53 54### Processing Details55 56- **Frame rate**: 5 FPS (downsampled from VPT's 20 FPS)57- **Action chunks**: 4 per frame (each 50ms = 200ms total)58- **Filtering**: Idle frames removed, loading screens filtered59 60## Usage61 62```python63from datasets import load_dataset64 65# Streaming (recommended - no download required)66ds = load_dataset("TESS-Computer/minecraft-vla-stage1", split="train", streaming=True)67 68for sample in ds:69 image = sample["image"] # PIL Image or bytes70 action = sample["action"]71 # Process...72```73 74## Training Pipeline75 76This is Stage 1 of a 3-stage training pipeline:771. **Stage 1** (this dataset): Action pretraining - learn observation→action mapping782. **Stage 2**: Instruction following - add task instructions from JARVIS-VLA793. **Stage 3**: Reasoning - add chain-of-thought before complex actions80 81## Citation82 83If you use this dataset, please cite:84- [OpenAI VPT](https://arxiv.org/abs/2206.11795) - Original contractor data85- [JARVIS-VLA](https://craftjarvis.github.io/JarvisVLA/) - Instruction annotations86- [Lumine](https://www.lumine-ai.org/) - Training methodology87 88## License89 90MIT License. Original VPT data is released under MIT by OpenAI.91