diffusers/matrix-game-2-modular

1# Matrix Game 2.0 Modular Pipeline2 3## Set Up4 5```shell6uv venv -p 3.107uv pip install -r requirements.txt8uv pip install git+https://github.com/huggingface/diffusers.git9```10 11## How to Use12 13```python14import torch15from diffusers import ModularPipelineBlocks16from diffusers.utils import export_to_video, load_image17from diffusers.modular_pipelines import WanModularPipeline18 19class MatrixGameWanModularPipeline(WanModularPipeline):20    """21    A ModularPipeline for MatrixGameWan.22 23    <Tip warning={true}>24 25        This is an experimental feature and is likely to change in the future.26 27    </Tip>28    """29 30    @property31    def default_sample_height(self):32        return 4433 34    @property35    def default_sample_width(self):36        return 8037 38# Download custom blocks for the pipeline39blocks = ModularPipelineBlocks.from_pretrained(40    "diffusers-internal-dev/matrix-game-2-modular",41    trust_remote_code=True,42)43 44# Initialize the pipeline runtime using the config in the repo45pipe = MatrixGameWanModularPipeline(blocks, "diffusers-internal-dev/matrix-game-2-modular")46 47# Load the model components of the pipeline48pipe.load_components(49    trust_remote_code=True,50    device_map="cuda",51    torch_dtype={"default": torch.bfloat16, "vae": torch.float32}52)53 54image = load_image("https://github.com/SkyworkAI/Matrix-Game/blob/main/Matrix-Game-2/demo_images/universal/0016.png?raw=true")55output = pipe(image=image, num_frames=141)56export_to_video(output.values['videos'][0], "matrix-game.mp4")57```58 59## Providing Actions as Inputs60 61Each action is represented as a string. The available actions are:62 63Motion Actions: ["forward", "left", "right"]64Camera Actions: ["camera_l", "camera_r", "camera_u", "camera_d"]65Compound Actions: Combinations of motion and camera actions with an `_` separating actions, e.g. "forward_left", "forward_left_camera_l"66 67```py68image = load_image("https://github.com/SkyworkAI/Matrix-Game/blob/main/Matrix-Game-2/demo_images/universal/0016.png?raw=true")69output = pipe(image=image, actions=["forward", "camera_l"], num_frames=141)70export_to_video(output.values['videos'][0], "matrix-game.mp4")71```