This repository provides the Real-World Reinforcement Learning (RWRL) Suite, a framework and set of environments designed to evaluate RL algorithms against challenges encountered in real-world applications. It targets RL researchers and practitioners seeking to bridge the gap between simulated and practical RL performance. The suite enables reproducible experimentation on factors like safety constraints, action/observation delays, noise, and non-stationary perturbations.

How It Works

The suite extends existing RL environments (Cartpole, Walker, Quadruped, Manipulator, Humanoid) with specific challenge wrappers. These wrappers inject realistic complexities such as delayed actions/observations, noisy inputs, dropped data, and physical perturbations. The framework standardizes evaluation by providing consistent logging and a multi-objective reward system, allowing for direct comparison of algorithm robustness across various challenging scenarios.

Quick Start & Requirements

• Install: pip3 install realworldrl_suite/
• Prerequisites: MuJoCo installation (see DeepMind's dm_control), Python 3.x, pip. PPO examples require TensorFlow 1.15.0, dm2gym, and OpenAI baselines. DMPO examples require dm-acme, dm-acme[reverb], dm-acme[tf], gym, jax, and dm-sonnet. MuJoCo license key required in ~/.mujoco/mjkey.txt.
• Links: Documentation, Examples

Highlighted Details

• Implements eight key challenges identified in "An Empirical Investigation of the Challenges of Real-World Reinforcement Learning."
• Offers combined challenge benchmarks with 'Easy', 'Medium', and 'Hard' difficulty levels.
• Includes example implementations for Random, PPO, and DMPO agents.
• Supports offline learning via the RL Unplugged library.

Maintenance & Community

• Developed by Google Research.
• Questions can be directed to realworldrl@google.com.
• Primary paper: An empirical investigation of the challenges of real-world reinforcement learning

Licensing & Compatibility

• The repository itself is not explicitly licensed in the README. However, it depends on MuJoCo (DeepMind), which has its own licensing. The code likely inherits Apache 2.0 or similar from Google Research, but this should be verified.

Limitations & Caveats

The Manipulator environment is noted as less tested. The PPO example requires TensorFlow 1.15.0, which is an older version. The specific license for the suite itself is not clearly stated, which could impact commercial use.