This repository provides the official PyTorch implementation of Heterogeneous-Agent Reinforcement Learning (HARL) algorithms, including HAPPO, HATRPO, HAA2C, HADDPG, HATD3, HAD3QN, and HASAC. It addresses the challenge of multi-agent cooperation in heterogeneous settings without parameter sharing, offering theoretical guarantees and superior performance on various benchmarks. The target audience includes researchers and practitioners in multi-agent reinforcement learning.

How It Works

HARL algorithms employ a sequential update scheme, contrasting with the simultaneous updates of MAPPO and MADDPG. This approach facilitates coordinated agent updates and is supported by theoretical guarantees for monotonic improvement and convergence to equilibrium. Both on-policy and off-policy variants are available, demonstrating effectiveness across diverse benchmarks.

Quick Start & Requirements

• Installation:

  conda create -n harl python=3.8
  conda activate harl
  # Install PyTorch >= 1.9.0 (CUDA >= 11.0) manually
  git clone https://github.com/PKU-MARL/HARL.git
  cd HARL
  pip install -e .
  

• Environment Dependencies: Requires installation for specific environments like SMAC, SMACv2, MAMuJoCo, MPE, Google Research Football, Bi-DexterousHands (requires IsaacGym), and Light Aircraft Game.
• Links: Official Documentation

Highlighted Details

• Implements seven HARL algorithms and interfaces for seven common MARL environments.
• Outperforms MAPPO, MADDPG, and MATD3 on MPE, MAMuJoCo, GRF, and Bi-DexterousHands benchmarks.
• HAPPO and HATRPO show comparable or better performance than MAPPO and QMIX in homogeneous settings (SMAC/SMACv2).
• Supports continuous, discrete, and multi-discrete action spaces for various algorithms.

Maintenance & Community

• Affiliated with Peking University and BIGAI.
• Includes citations for JMLR 2024 and ICLR 2024 spotlight papers.

Licensing & Compatibility

• The repository itself is not explicitly licensed in the README. However, it depends on other libraries with their own licenses. Users should verify compatibility for commercial or closed-source use.

Limitations & Caveats

• Bi-DexterousHands requires IsaacGym, which has specific hardware requirements.
• Installation of some environments (e.g., MuJoCo, SMAC) involves external dependencies and specific setup steps.
• The Light Aircraft Game environment currently only supports self-play for 2v2 settings, with custom cooperative tasks implemented.