EnvPool is a C++-based parallel execution engine designed to significantly accelerate Reinforcement Learning (RL) environment interactions. It targets RL researchers and practitioners seeking to boost simulation throughput beyond standard Python-based vectorization, offering substantial speedups for a wide range of RL environments.

How It Works

EnvPool leverages C++ for core environment execution and pybind11 for Python integration, utilizing a thread pool for parallel processing. This architecture allows for batched interactions with multiple environment instances, supporting both synchronous and asynchronous execution models. The C++ backend minimizes Python overhead, enabling high-performance simulation, with a notable advantage in CPU-bound environments where Python's GIL can be a bottleneck.

Quick Start & Requirements

• Install: pip install envpool
• Prerequisites: Python >= 3.7.
• Documentation: envpool.readthedocs.io
• Examples: examples/

Highlighted Details

• Achieves up to 1M raw FPS for Atari and 3M raw FPS for Mujoco on high-end hardware (256 CPU cores).
• Provides ~20x throughput improvement over gym.vector_env on high-end setups and ~3x on typical PCs (12 CPU cores).
• Compatible with OpenAI Gym, DeepMind dm_env, and Gymnasium APIs.
• Supports synchronous, asynchronous, single-player, and multi-player environment execution.
• Offers an XLA interface for JAX integration.

Maintenance & Community

• The project is actively developed, with contributions welcomed.
• Integration examples are provided for popular RL libraries like Stable-Baselines3, Tianshou, ACME, and CleanRL.

Licensing & Compatibility

• Licensed under Apache 2.0.
• Compatible with commercial use and closed-source linking.

Limitations & Caveats

• While it supports many environments, adding custom C++ environments requires developer effort.
• Performance gains are most pronounced on multi-core systems; benefits on single-core setups may be less significant.