Matrix-Game is a 17B-parameter foundation model for generating controllable game worlds, specifically targeting interactive video generation conditioned on user inputs. It offers fine-grained control over dynamic scene evolution and includes a benchmark for evaluating Minecraft world models.

How It Works

This project utilizes a diffusion-based image-to-world model that generates high-quality videos based on keyboard and mouse inputs. This approach allows for dynamic scene evolution and precise user control, differentiating it from static world generation methods.

Quick Start & Requirements

• Install dependencies: pip install -r requirements.txt
• Install Apex and FlashAttention-3.
• Inference: bash run_inference.sh
• Hardware: Requires NVIDIA A100/H100 GPU with ≥80GB VRAM for single inference.

Highlighted Details

• Outperforms prior models in human evaluations across Overall Quality, Controllability, Visual Quality, and Temporal Consistency.
• Achieves superior scores on the GameWorld benchmark, particularly in Keyboard Accuracy (0.95) and Mouse Accuracy (0.95).
• Features a large-scale Minecraft dataset with fine-grained action annotations for scalable training.
• Supports parallel inference scripts for improved efficiency.

Maintenance & Community

The project is associated with SkyworkAI. Further community or roadmap details are not explicitly provided in the README.

Licensing & Compatibility

Licensed under the MIT License, permitting commercial use and integration with closed-source projects.

Limitations & Caveats

The model has significant hardware requirements, demanding at least 80GB of VRAM on an A100/H100 GPU for inference.