aoiandroid/NitroGen

1---2license: other3datasets:4- nvidia/NitroGen5tags:6- behavior7- cloning8- gaming9- agent10pipeline_tag: reinforcement-learning11---12 13<img src="https://cdn-uploads.huggingface.co/production/uploads/67d8509cb6b70254852d734d/u3VY6_KoT6tEs86YPehU2.gif" width="100%" />14 15<div align="center">16  <p style="font-size: 1.2em;">17    <a href="https://nitrogen.minedojo.org/"><strong>Website</strong></a> | 18    <a href="https://github.com/MineDojo/NitroGen"><strong>Code</strong></a> |19    <a href="https://huggingface.co/datasets/nvidia/NitroGen"><strong>Dataset</strong></a> |20    <a href="https://huggingface.co/papers/2601.02427"><strong>Paper</strong></a>21  </p>22</div>23 24# NitroGen: An Open Foundation Model for Generalist Gaming Agents25 26NitroGen is a unified vision-to-action foundation model designed to play video games directly from raw frames. It is a generalist agent trained via large-scale behavior cloning on 40,000 hours of gameplay across over 1,000 games. It maps RGB video footage to gamepad actions.27 28NitroGen works best on games designed for gamepad controls (e.g., action, platformer, and racing games) and is less effective on games that rely heavily on mouse and keyboard (e.g., RTS, MOBA).29 30## Sample Usage31 32### Installation33 34To use NitroGen, clone and install the repository:35 36```bash37git clone https://github.com/MineDojo/NitroGen.git38cd NitroGen39pip install -e .40```41 42### Inference43 441. **Download the checkpoint** from Hugging Face:45```bash46hf download nvidia/NitroGen ng.pt47```48 492. **Start the inference server**:50```bash51python scripts/serve.py <path_to_ng.pt>  52```53 543. **Run the agent** on the game of your choice (currently supports Windows games):55```bash56python scripts/play.py --process '<game_executable_name>.exe'57```58 59## Model Details60 61- **Architecture:** Vision Transformer (SigLip2) + Diffusion Matching Transformer (DiT).62- **Parameters:** $4.93 \times 10^8$.63- **Inputs:** 256x256 RGB images.64- **Outputs:** Gamepad actions (21x16 shape: two 2D continuous vectors for joysticks, 17 binary buttons).65- **Training:** Trained on 40,000 hours of internet-scale gameplay videos.66 67## Citation68 69If you find NitroGen useful in your research, please cite:70 71```bibtex72@misc{magne2026nitrogen,73      title={NitroGen: An Open Foundation Model for Generalist Gaming Agents}, 74      author={Loïc Magne and Anas Awadalla and Guanzhi Wang and Yinzhen Xu and Joshua Belofsky and Fengyuan Hu and Joohwan Kim and Ludwig Schmidt and Georgia Gkioxari and Jan Kautz and Yisong Yue and Yejin Choi and Yuke Zhu and Linxi "Jim" Fan},75      year={2026},76      eprint={2601.02427},77      archivePrefix={arXiv},78      primaryClass={cs.CV},79      url={https://arxiv.org/abs/2601.02427}, 80}81```82 83## License84 85Governing Terms: [NVIDIA License](https://developer.download.nvidia.com/licenses/NVIDIA-OneWay-Noncommercial-License-22Mar2022.pdf).86The model uses a [SigLip2](https://huggingface.co/google/siglip2-base-patch16-224) backbone which is licensed under Apache 2.0.