Summary

torch-twrl is a reinforcement learning (RL) framework developed in Lua/Torch by Twitter. It provides researchers and practitioners with tools to implement and experiment with various RL algorithms, integrating seamlessly with the popular OpenAI Gym environments through a dedicated HTTP API. The framework aims to simplify RL development within the Torch ecosystem.

How It Works

The core of torch-twrl lies in its modular design, allowing agents to be defined by combinations of models (e.g., MLP), policies (e.g., categorical, stochastic), and learning updates (e.g., reinforce, tdLambda). It bridges the gap between Lua/Torch and Python-based OpenAI Gym environments using a gym-http-api submodule, which exposes Gym functionalities over HTTP. This approach enables RL agents written in Lua to interact with a wide array of simulated environments.

Quick Start & Requirements

Installation involves setting up the Torch scientific computing framework, followed by torch-twrl itself.

• Torch: Clone torch/distro, run bash install-deps, then ./install.sh.
• torch-twrl: Clone the repository, cd into it, and run luarocks make.
• OpenAI Gym: Requires Python 3, virtualenv, pip install gym, and pip install -r src/gym-http-api/requirements.txt. ffmpeg is also needed for rendering.
• Running: Start the gym_http_server.py and then execute example scripts like cartpole-pg.sh.

Highlighted Details

• Implements key RL algorithms including TD(Lambda) with eligibility traces and Policy Gradient (Reinforce) using MLPs.
• Full compatibility with OpenAI Gym environments via the gym-http-api submodule.
• Comprehensive testing suite using Travis CI, supporting multiple Lua versions (LUAJIT21, LUA51, LUA52) and compilers (gcc, clang).
• Cites foundational RL research papers, indicating a theoretically grounded approach.

Maintenance & Community

The README does not provide specific details regarding active maintenance, contributor statistics, or community channels like Discord or Slack.

Licensing & Compatibility

torch-twrl is released under the permissive MIT License, allowing for broad compatibility with commercial and closed-source projects.

Limitations & Caveats

A known issue exists where tilecoding examples fail in LUA52 due to incompatibility with libhash. Several advanced RL algorithms like DQN, A3C, and DDPG are listed under "Future Work," suggesting they are not yet implemented or are in early development stages. The project's last commit date (implied by the 2016 copyright) may indicate limited recent activity.