proj-airi/game-playing-ai-playground-2d

2D Game Playing AI Playground(WIP)

This is a playground for some models that can play 2D games.

• —Deployed on HuggingFace space.
• —Powered by ONNX runtime.

Usage

• —Upload an image[^1] to detect objects in the image.
• —Upload a local video to detect objects while it plays.
• —Connect to your VNC server to play games and detect objects in real-time.

[^1]: Square image is required when using Factorio YOLO v0 model.

There are some methods to run object detection model in real-time.

• —Run game client in Docker
• —TODO: Capture game screen
• —Upload video file

The playground currently supports raw YOLO outputs that require external NMS and YOLO26 end-to-end outputs shaped as (1, max_det, 6). Model cards may declare the format in their YAML frontmatter:

playground:
  dataset_revision: <dataset-commit>
  input_size: 640
  output_format: yolo26-end-to-end

Supported values are yolo-raw and yolo26-end-to-end. Existing model cards without output_format use the format registered by the playground for backward compatibility.

Run Game Client in Docker

Please follow the instructions below to run game client in Docker.

<details> <summary>Factorio</summary>

Because of the license of Factorio, we cannot provide pre-built Factorio docker image, you need to build it yourself.

First, clone the repository:

git clone https://github.com/moeru-ai/airi-factorio.git
cd airi-factorio/docker

Then, build the docker image, you can find the version history here:

docker build . -t factorio-gui --build-arg factorio_login='<USERNAME>' --build-arg factorio_pwd='<PASSWORD>' --build-arg factorio_version=<VERSION>

Then, run the docker container:

docker run -it -p 5900:5900 -p 5901:5901 factorio-gui

After that, you can visit http://localhost:5901/vnc.html to play Factorio. </details>

You need to type ws://localhost:5901/websockify in the WebSocket URL field, and then click Connect, after connected, you can play the game and view detection result in real-time.