This repository provides the source code for the ICLR 2024 paper "Building Cooperative Embodied Agents Modularly with Large Language Models." It enables the creation of modular, LLM-powered embodied agents capable of complex cooperative tasks in simulated environments, targeting researchers and developers in embodied AI and multi-agent systems.

How It Works

The project leverages large language models (LLMs) to control embodied agents, breaking down complex tasks into modular components. Agents interact within simulated environments like ThreeDWorld Multi-Agent Transport (TDW-MAT) and Communicative Watch-And-Help (C-WAH), utilizing LLMs for decision-making, communication, and task execution. This modular LLM-driven approach facilitates emergent cooperative behaviors and simplifies the development of sophisticated embodied AI.

Quick Start & Requirements

• TDW-MAT: Requires Python 3.9, PyTorch, and TDW platform setup. Installation involves pip install -e . within the tdw_mat directory. A demo scene can be run with python demo/demo_scene.py.
• C-WAH: Requires Python 3.8, VirtualHome simulator, and API. Installation involves cloning the VirtualHome repository, downloading the simulator, and running pip install -r requirements.txt within the cwah directory.
• Dependencies: Specific LLM models (e.g., GPT-4) are used for agent control, requiring API access. TDW setup may need an X server on remote Linux machines.

Highlighted Details

• Implements two distinct multi-agent environments: TDW-MAT for object transport and C-WAH for household chores.
• TDW-MAT features configurable settings (Enough Container vs. Rare Container) and metrics like Transport Rate (TR) and Efficiency Improvements (EI).
• C-WAH supports five household tasks with predicate-based subgoals and metrics like Average Steps (L) and EI.
• Includes example scripts for running experiments with baseline and CoELA agents.

Maintenance & Community

The project is associated with UMass-Embodied-AGI and has recent updates addressing navigation issues in TDW-MAT. The paper was published in ICLR 2024.

Licensing & Compatibility

The repository itself does not explicitly state a license. The included VirtualHome repository is under the MIT license. Compatibility for commercial use depends on the licenses of the underlying LLMs and simulators used.

Limitations & Caveats

The setup for both environments can be complex, requiring specific simulator versions and configurations. The reliance on external LLM APIs (like GPT-4) introduces potential costs and dependencies. The project is presented as research code, and stability for production use is not guaranteed.