This repository implements the Hierarchical Actor-Critic (HAC) algorithm, designed to accelerate reinforcement learning for agents by enabling them to decompose complex tasks into simpler, sequential sub-goals. It is targeted at researchers and practitioners in reinforcement learning seeking to improve sample efficiency and learning speed in complex robotic manipulation and control tasks.

How It Works

HAC employs a hierarchical reinforcement learning approach, where multiple actor-critic agents are stacked in a hierarchy. Higher-level agents set goals for lower-level agents, which then learn policies to achieve those goals. This structured decomposition allows for more efficient exploration and faster learning compared to traditional flat RL methods. The implementation features bounded Q-values for improved critic stability and offers the option to disable target networks, a deviation from standard DDPG, which has shown performance benefits.

Quick Start & Requirements

• Primary install / run command: python3 initialize_HAC.py --retrain
• Prerequisites: MuJoCo license, MuJoCo software libraries, MuJoCo Python wrapper from OpenAI.
• The UR5 agent with a 3-level hierarchy is expected to achieve 90+% success in ~350 episodes.
• Official paper: ICLR 2019 paper
• Blog post: Blog post

Highlighted Details

• Implements a 3-level hierarchy for UR5 reacher tasks, achieving high success rates.
• Includes environments for UR5 reacher, inverted pendulum, and ant domains (reacher and four rooms).
• Features a centralized design template (design_agent_and_env.py) for configuring agent and environment hyperparameters.
• Offers options for bounded Q-values and disabling target networks for improved critic performance.

Maintenance & Community

• Last update log entry: 5/20/2020, adding 2-level ant environments and centralizing exploration hyperparameters.
• Contact: andrew_levy2@brown.edu

Licensing & Compatibility

• License: Not explicitly stated in the README.
• Compatibility: Requires specific MuJoCo dependencies, which may have licensing implications for commercial use.

Limitations & Caveats

The code for ant domains is noted as temporarily added and will be integrated in the future. The README does not specify the exact license, which could impact commercial adoption.