NikhilKhatri/Meta-LLM-Combat-Agent

Unreal Engine + LLM Combat Agent

This space runs a headless Unreal Engine 5 environment controlled by a fine-tuned Qwen LoRA model.

⚔️ Meta-LLM Combat Agent: Unreal Engine 5 x OpenEnv

  [](https://youtu.be/hXlIr6uVKyI) 

TL;DR: We built a fully decoupled microservice architecture that runs a headless Unreal Engine 5 combat game inside a cloud Docker container, translated through OpenEnv, and controlled by a fine-tuned Qwen2.5-0.5B Large Language Model.

🎯 The Problem: Replacing Traditional AI with LLMs

For decades, video game AI has relied on rigid Behavior Trees and Finite State Machines. Traditional NPC enemies follow predictable if-then scripts, making games repetitive and easy to exploit.

Our environment asks a frontier question: Can an LLM replace traditional game code and learn real-time 3D spatial combat purely through Reinforcement Learning?

By moving from hardcoded scripts to generalized LLM reasoning, game developers can create dynamic, unpredictable, and highly adaptive enemies that learn how to fight based on physical rewards rather than pre-programmed rules.

🏗️ Environment Innovation & Architecture

This is not a turn-based grid or a text adventure. This is a live, physics-based 3D environment. To achieve this in the cloud, we engineered a custom pipeline:

1. 1.The Body (Hugging Face Space): Unreal Engine 5 runs inside a headless Docker container. We bypass traditional Vulkan graphics requirements using the -nullrhi flag, allowing the 3D physics and combat math to run entirely on the CPU.
2. 2.The Translator (OpenEnv + Schola): UE5 communicates via the AMD Schola plugin (TCP/Protobuf). We use the OpenEnv server as a Universal Translator, catching WebSockets from the internet on Port 7860 and translating them into raw TCP commands for the game engine.
3. 3.The Brain (Google Colab/Local): A Python client reads the OpenEnv observations over the internet and feeds them to our LLM, which sends back JSON keystrokes in real-time.

🚀 Quick Start & Setup Guide

Phase 1: Running the Game Engine (Server)

The environment is containerized and ready to deploy on Hugging Face Spaces.

1. 1.Clone this repository to your local machine or Hugging Face Space.
2. 2.Ensure you have the Unreal Engine Linux build (LinuxBuild.zip) in the root directory.
3. 3.The included Dockerfile handles installing the specific older versions of gym==0.21.0 and setuptools required to bypass PEP-517 build isolation issues with the Schola library.
4. 4.The start.sh script automatically launches UE5 in headless mode and binds the OpenEnv Translator to public port 7860:

/app/GameData/Linux/AIEnemyFightEnv.sh -nullrhi -NoSound -stdout &
uvicorn envs.free_guy.server.app:app --host 0.0.0.0 --port 7860

Phase 2: Running the LLM Agent (Client)

You can run the agent locally or via Google Colab.

1. 1.Open the Google Colab Training Script.

1. 1.Install the OpenEnv client and Unsloth:

pip install openenv unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git

1. 1.Update the  in the Colab notebook to point to your live Hugging Face Space (e.g., https://nikhilkhatri-meta-llm-combat-agent.hf.space).

1. 1.Run the cell! The Colab notebook will download our fine-tuned LoRA model, connect to the cloud engine, and begin live combat.

🎮 Observation & Action Space

Observation State (Input to LLM)

The LLM receives a dense text string detailing the spatial reality: State: Distance 6.64m, Your Health 3.0, Enemy Health 3.0, Enemy Flanking

Action Space (Output from LLM)

The LLM must output a strict JSON array representing physical keyboard presses:

{"W": 1.0, "A": 0.0, "S": 0.0, "D": 0.0, "Attack": 1.0}

⚖️ The Reward Signal

To prevent "Reward Hiding" (where the AI learns to stand still to avoid being penalized for missing attacks), our reward rubric is strictly logical:

• —Attack when close (< 2m): Massive positive reward (+0.96).
• —Stand still when close: Massive negative penalty (-1.22).
• —Swing at the air when far (> 2m): Negative penalty (-0.30).
• —Close the distance (W=1.0) when far: Positive progression reward (+0.20).

🧠 Training & Results (Offline RL)

We utilized a Qwen2.5-0.5B-Instruct model. Initially, the base model hallucinated text responses or chose "lazy" static actions to avoid harsh penalties.

Using Unsloth and Hugging Face TRL (SFTTrainer), we implemented an Offline RL pipeline (Filtered Behavioral Cloning). We ran the base model through thousands of simulated episodes, captured the raw Schola rewards using a custom OpenEnv wiretap, filtered out catastrophic failures (reward < -0.4), and trained the LoRA adapters exclusively on actions that maximized survival and damage.

📈 Evidence of Learning: Before & After

Before Training: The Base Model

The untrained Qwen2.5-0.5B hallucinates or stands still, resulting in heavy negative rewards. The blue trendline is trapped below the survival baseline.

After Training: The Fine-Tuned Agent

After offline RL fine-tuning, the agent consistently closes the distance and executes attacks. The 5-episode moving average (blue line) breaks through the survival baseline into positive rewards.

Uncaging the LLM: Training a Qwen Combat Agent in Unreal Engine 5

Introduction

When we play video games, we are rarely fighting "intelligence." We are fighting Behavior Trees—massive, hardcoded flowcharts written by developers. If Player is X meters away, then Execute Attack Y. Once you learn the pattern, the game is solved.

For the OpenEnv Hackathon, we wanted to ask a bigger question: What happens if we rip out the hardcoded game logic, plug an LLM directly into a live AAA game engine, and let it figure out how to fight using Reinforcement Learning?

The result is the Meta-LLM Combat Agent, an environment where Unreal Engine 5 and a Qwen2.5-0.5B model clash across a decoupled cloud architecture.

The Engineering: Building the Bridge

The hardest part of training an AI in 3D space isn't the AI—it's the plumbing. You cannot easily run a massive desktop game engine inside a standard Linux Docker container.

To make our environment accessible for judges and researchers, we containerized Unreal Engine 5 on a Hugging Face Space. By utilizing the -nullrhi boot flag, we completely disabled the Vulkan graphics pipeline, allowing the physics, collision, and combat math to run headless on the CPU.

We then deployed OpenEnv as our Universal Translator. Unreal Engine speaks a strict TCP/Protobuf language via the AMD Schola plugin. OpenEnv sits in the middle, accepts WebSockets from our Python inference scripts over the public internet, translates the data, and feeds it into the game.

The Challenge: Herding a Drunk AI

Our agent's "eyes" consist of a simple text string: Distance 6.64m, Your Health 3.0, Enemy Health 3.0, Enemy Flanking. Its "hands" are a JSON array of keyboard inputs.

When we first connected Qwen2.5-0.5B to the game, it had a total existential crisis. Instead of fighting, it outputted text like: "This is my first time playing the game. Please help me understand what I should do." Even when we forced it to output JSON, the agent quickly discovered "Reward Hiding." It realized that running blindly into the enemy resulted in a massive -1.22 damage penalty, but standing perfectly still only gave a -0.30 penalty. The AI became a lazy gamer, refusing to touch the keyboard.

The Solution: Offline RL and Smooth Curves

To fix this, we generated thousands of synthetic and exploratory data scenarios. We wrote a custom Python wiretap to extract the hidden floating-point rewards from the OpenEnv data stream.

Using Unsloth and TRL, we passed the data through an Offline Reinforcement Learning filter. We threw away every scenario where the AI took massive damage, and mathematically forced the model's weights to mimic the scenarios where it successfully closed the distance and attacked.

The result is visible in our training curves. In our raw data, the rewards fluctuate wildly. But when you apply a 5-episode moving average, a beautiful trend emerges: the post-trained AI aggressively breaks through the negative survival baseline, stops hallucinating text, and consistently achieves the maximum +0.96 reward for landing close-quarters combat strikes.

Why This Matters

We proved that OpenEnv can be used to bridge standard LLM training pipelines with highly complex, industry-standard 3D game engines. By replacing rigid state machines with trainable LLMs, the future of gaming isn't just about better graphics—it's about enemies that actually think, learn, and adapt to how you play.