AI-Wukong is a Reinforcement Learning (RL) framework designed for playing Action RPGs (ARPGs) like "Black Myth: Wukong." It targets researchers and developers interested in AI game agents, offering a hybrid approach that combines RL for combat with large language models (LLMs) for exploration, aiming for improved performance over pure RL or LLM solutions.

How It Works

The framework employs a modular design, decoupling core functionalities like visual perception, decision-making, and exploration. Combat relies on RL algorithms (DQN/PPO) trained with a ResNet-based vision model to process game frames, identify enemy states, and execute actions. Exploration and interaction leverage multimodal LLMs, with a switch to the RL combat module when engagement begins. This separation allows for independent iteration and replacement of modules.

Quick Start & Requirements

• Install: Clone the repository, create a conda environment (conda create --name RL-ARPG-dev python=3.10), activate it (conda activate RL-ARPG-dev), and install dependencies (pip install -r requirements.txt).
• Prerequisites: Python >= 3.10, Tensorflow-gpu 1.15.2, Pytorch 2.3.1, OpenCV, CUDA >= 11.8.0, CuDNN >= 7.6.0. Requires game installation via Steam/Wegame. LLM API keys (OpenAI, Claude) are needed for the exploration module.
• Configuration: Game resolution set to 1680x1050 windowed, positioned at the top-left. Specific keybindings are suggested (J: light attack, M: heavy attack, O: sprint, K: dodge).
• Links: GitHub Repository, Demo Video

Highlighted Details

• Combines DQN/PPO RL with ResNet vision models for combat.
• Utilizes LLMs (GPT-4o, Claude) for map exploration and interaction.
• Decoupled modules for combat and exploration, enabling independent development.
• Combat module reaction time can reach 0.2s.
• Vision module has limitations recognizing non-humanoid enemies.

Maintenance & Community

The project is developed by students from Shanghai Jiao Tong University, Tokyo Institute of Technology, and Beijing University of Posts and Telecommunications. The latest development log entries are available in the README.

Licensing & Compatibility

The project is intended for technical research and popular science purposes only and does not grant commercial application authorization. The specific license is not explicitly stated but the disclaimer suggests non-commercial use.

Limitations & Caveats

The vision module exhibits poor recognition of non-humanoid enemies. Configuration of game resolution and UI elements is necessary for non-standard display setups. The project is presented as a research demo, not a production-ready system.