jbilcke-hf/NON_WORKING_matrix_game

<!-- markdownlint-disable first-line-h1 --> <!-- markdownlint-disable html --> <!-- markdownlint-disable no-duplicate-header -->

Matrix-Game: Interactive World Foundation Model

<font size=7><div align='center' > [🤗 Huggingface] [📖 Technical Report] [🚀 Project Website] </div></font>

<div align="center"> <img src="assets/videos/demo.gif" alt="teaser" /> </div>

📝 Overview

Matrix-Game is a 17B-parameter interactive world foundation model for controllable game world generation.

✨ Key Features

• —🎯 Feature 1: Interactive Generation. A diffusion-based image-to-world model that generates high-quality videos conditioned on keyboard and mouse inputs, enabling fine-grained control and dynamic scene evolution.
• —🚀 Feature 2: GameWorld Score. A comprehensive benchmark for evaluating Minecraft world models across four key dimensions, including visual quality, temporal quality, action controllability, and physical rule understanding.
• —💡 Feature 3: Matrix-Game Dataset A large-scale Minecraft dataset with fine-grained action annotations, supporting scalable training for interactive and physically grounded world modeling.

🔥 Latest Updates

• —[2025-05] 🎉 Initial release of Matrix-Game Model

🚀 Performance Comparison

GameWorld Score Benchmark Comparison

Metric Descriptions:

• —Image Quality / Aesthetic: Visual fidelity and perceptual appeal of generated frames
• —Temporal Consistency / Motion Smoothness: Temporal coherence and smoothness between frames
• —Keyboard Accuracy / Mouse Accuracy: Accuracy in following user control signals
• —3D Consistency: Geometric stability and physical plausibility over time

Please check our GameWorld benchmark for detailed implementation.

Human Evaluation

[image]

Double-blind human evaluation by two independent groups across four key dimensions: Overall Quality, Controllability, Visual Quality, and Temporal Consistency. Scores represent the percentage of pairwise comparisons in which each method was preferred. Matrix-Game consistently outperforms prior models across all metrics and both groups.

🚀 Quick Start

# clone the repository:
git clone https://github.com/SkyworkAI/Matrix-Game.git
cd Matrix-Game

# install dependencies:
pip install -r requirements.txt

# install apex and FlashAttention-3
# Our project also depends on [apex](https://github.com/NVIDIA/apex) and [FlashAttention-3](https://github.com/Dao-AILab/flash-attention)

# Run batch inference to generate videos
bash run_inference.sh

# Run interactive websocket server
python server.py --model_root ./models/matrixgame

Interactive WebSocket Server

We've implemented a real-time interactive WebSocket server that uses the Matrix-Game model to generate game frames based on keyboard and mouse inputs:

Features:

• —Real-time Generation: Frames are generated on-the-fly based on user inputs
• —Keyboard & Mouse Control: Move through the virtual world using WASD keys and mouse movements
• —Multiple Scenes: Choose from different environments (forest, desert, beach, hills, etc.)
• —Fallback Mode: Automatically falls back to demo mode when GPU resources are unavailable

Usage:

# Basic startup
python server.py

# With custom model paths
python server.py --model_root ./models/matrixgame --port 8080

# With individual model component paths
python server.py --dit_path ./custom/dit --vae_path ./custom/vae --textenc_path ./custom/textenc

Connection:

• —WebSocket endpoint: ws://localhost:8080/ws
• —Web client: http://localhost:8080/

System Requirements:

• —NVIDIA GPU with CUDA support
• —24GB+ VRAM recommended for smooth frame generation

🔧 Hardware Requirements

• —GPU:
• —NVIDIA A100/H100
• —VRAM:
• —Requires ≥80GB of GPU memory for a single 65-frame video inference.

⭐ Acknowledgements

We would like to express our gratitude to:

• —Diffusers for their excellent diffusion model framework
• —HunyuanVideo for their strong base model
• —MineDojo for their Minecraft video dataset
• —MineRL for their excellent gym framework
• —Video-Pre-Training for their accurate Inverse Dynamics Model
• —GameFactory for their idea of action control module

We are grateful to the broader research community for their open exploration and contributions to the field of interactive world generation.

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

📎 Citation

If you find this project useful, please cite our paper:

@article{zhang2025matrixgame,
  title     = {Matrix-Game: Interactive World Foundation Model},
  author    = {Yifan Zhang and Chunli Peng and Boyang Wang and Puyi Wang and Qingcheng Zhu and Zedong Gao and Eric Li and Yang Liu and Yahui Zhou},
  journal   = {arXiv},
  year      = {2025}
}