D4RL is a benchmark suite for offline reinforcement learning, providing standardized environments and datasets for training and evaluating RL algorithms. It targets researchers and practitioners in RL, offering a consistent framework to compare algorithm performance and reproducibility.

How It Works

D4RL integrates with the OpenAI Gym API, allowing users to create environments using gym.make. Each environment is associated with a pre-collected offline dataset, accessible via env.get_dataset(). This dataset includes observations, actions, rewards, and termination flags. An alternative d4rl.qlearning_dataset function formats data for Q-learning, adding next_observations. Datasets are automatically downloaded and cached.

Quick Start & Requirements

• Install via pip: pip install git+https://github.com/Farama-Foundation/d4rl@master#egg=d4rl
• Control environments require MuJoCo and mujoco_py (license and setup needed).
• Flow and CARLA tasks have additional installation steps and dependencies.
• See official quick-start and task list.

Highlighted Details

• Provides a standardized API consistent with OpenAI Gym.
• Datasets are automatically downloaded and cached locally (~/.d4rl/datasets).
• Includes a get_normalized_score function for evaluating agent performance against reference scores.
• Aggregated implementations of offline RL algorithms are available in a separate repository.

Maintenance & Community

The project is undergoing a significant transition: environments are being moved to Gymnasium, MiniGrid, and Gymnasium-Robotics, and datasets to Minari. PyBullet and Flow environments are not planned for maintenance. Further details on the transition are in a blog post.

Licensing & Compatibility

Code is licensed under Apache 2.0. Datasets are licensed under Creative Commons Attribution 4.0 License (CC BY). Compatible with commercial use.

Limitations & Caveats

The project is actively migrating its components to newer, more actively maintained libraries (Gymnasium, Minari). Users should be aware that D4RL itself may receive limited future development, with new work focused on the successor libraries.