Ghost in the Minecraft (GITM) provides a framework for creating generally capable AI agents in the open-world environment of Minecraft, leveraging Large Language Models (LLMs) with text-based knowledge and memory. It aims to overcome the limitations of traditional RL-based agents in handling long-horizon, complex tasks, offering broad task coverage and significantly improved success rates.

How It Works

GITM employs a hierarchical LLM-based approach, decomposing goals into sub-goals, then structured actions, and finally keyboard/mouse operations. This contrasts with direct RL mapping. The framework includes an LLM Decomposer for goal breakdown using internet-sourced knowledge, an LLM Planner for action sequencing and memory integration, and an LLM Interface for environment interaction. This LLM-centric design allows for efficient learning and adaptation in complex, uncertain environments.

Quick Start & Requirements

• Install/Run: Not explicitly detailed in the README, but likely involves Python and interaction with the Minecraft environment.
• Prerequisites: The project highlights its ability to train on a single CPU node with 32 CPU cores, requiring no GPUs.
• Resources: Training is stated to take 2 days on a single CPU node.
• Links: Arxiv Paper, Demo Video

Highlighted Details

• Achieves 100% completion of the Minecraft Overworld technology tree, compared to 30% for prior combined methods.
• Reports a 67.5% success rate for the "ObtainDiamond" task, a +47.5% improvement over OpenAI's VPT.
• Demonstrates exceptional training efficiency, requiring only 2 days on a 32-core CPU, vastly outperforming GPU-intensive methods.
• Handles diverse biomes, environments, day/night cycles, and monster encounters.

Maintenance & Community

• The project is associated with OpenGVLab. No specific community links (Discord, Slack) or roadmap are provided in the README.

Licensing & Compatibility

• The README does not specify a license. Citation is provided for academic use.

Limitations & Caveats

• The README does not detail specific limitations, unsupported platforms, or known bugs. The primary focus is on the capabilities and efficiency of the LLM-based approach.