OpenPipe ART is an open-source library designed to enhance the performance of multi-turn LLM agents through Reinforcement Learning (RL), specifically using the GRPO algorithm. It targets developers and researchers looking to fine-tune LLMs for agentic workflows with minimal code modification, offering a streamlined approach to RL training.

How It Works

ART employs a client-server architecture. The client integrates with existing agent codebases, routing LLM requests to the ART server. The server, running independently, manages model inference (via vLLM with LoRA) and the GRPO training loop. It collects agent trajectories, assigns rewards, trains the model, and updates the inference endpoint, creating a continuous feedback loop for agent improvement. This separation allows users to focus on agent logic while ART handles the RL complexities.

Quick Start & Requirements

• Install: pip install openpipe
• Prerequisites: Python 3.8+, vLLM-compatible models (e.g., HuggingFace Transformers, Unsloth-optimized models). Gemma 3 is noted as unsupported. GPU with sufficient VRAM for model inference and training is required.
• Resources: Setup involves integrating the ART client into an existing agent and running the ART server. Specific resource requirements depend on the LLM size and training data.
• Docs: Notebooks

Highlighted Details

• Trains LLMs using the GRPO reinforcement learning algorithm.
• Integrates with existing agent codebases via an OpenAI-compatible client.
• Leverages vLLM for efficient inference and Unsloth for potential performance gains.
• Supports fine-tuning with LoRA adapters.

Maintenance & Community

ART is under active development with contributions welcomed. Community interaction is encouraged via Discord.

Licensing & Compatibility

• License: Apache-2.0 License.
• Compatibility: Permissive license allows for commercial use and integration with closed-source applications.

Limitations & Caveats

Gemma 3 models are explicitly listed as unsupported. The library is in active development, suggesting potential for breaking changes or evolving features.