Scalable RL for Any Agent and Sandbox

Dressage is a scalable RL framework designed for training LLM agents that interact with real-world tools. It addresses the challenge of bridging policy rollouts, sandboxed tool execution, and training data conversion, enabling diverse agents to leverage full RL gradient flow. The framework supports both whitebox (Python) and blackbox (HTTP) agent paradigms through a unified interface, offering a robust solution for advanced agent development.

How It Works

Built atop the slime RL framework, Dressage employs a token-level inference proxy for granular trajectory recording using its Token-In-Token-Out (TITO) mechanism. This approach avoids retokenization drift and enables precise control over generation, pause/resume, and MoE routing. A unified "paddock" layer abstracts environment interaction, supporting diverse sandboxes like local bubblewrap or remote E2B. This separation of concerns allows flexible mixing of agent types (whitebox/blackbox) and execution environments (local/remote) without modifying agent logic, while slime-native integration ensures seamless extension.

Quick Start & Requirements

Installation is facilitated via Docker (docker pull huang3eng/dressage:v0.1.0, docker run --rm --gpus all --ipc=host --shm-size=16g ...). Key prerequisites include NVIDIA GPUs, sufficient shared memory, and specific model checkpoints (e.g., Qwen/Qwen3.5-4B) prepared using provided scripts. The framework integrates with slime for training orchestration. Detailed setup, configuration, and troubleshooting are available in the official Quick Start Guide and Whitebox Agent Quick Start documentation.

Highlighted Details

• Any Agent, Any Sandbox: Provides a unified interface for whitebox Python agents, blackbox HTTP agents (e.g., opencode, openclaw), and various sandbox providers including local bubblewrap and remote E2B.
• Token-Wise Control (TITO): Captures training evidence at token granularity (token_id, logprob, loss_mask, token_version, token_expert), enabling advanced features like incremental tokenization, version-aware masking, and MoE routing replay.
• Segment-Aware Training: Expands split trajectory segments into training samples, broadcasting anchor segment advantages and employing prompt-equal denominators for equitable gradient weighting.
• Production-Grade Rollout Safety: Implements atomic trajectory logging, HTTP error redaction, context overflow detection, and session artifact archiving for robust and diagnosable rollout execution.

Maintenance & Community

Dressage is developed by the Alibaba Accio team, with core contributors listed for technical inquiries. Contributions are welcomed via GitHub Issues and Pull Requests. Specific contact points are provided for academic collaborations and technical discussions.

Licensing & Compatibility

The project is licensed under the permissive Apache License 2.0, facilitating commercial use. However, users must review the upstream licenses for bundled tools like opencode and openclaw, as they are not distributed by this repository.

Limitations & Caveats

The Token-In-Token-Out (TITO) implementation currently supports only the Qwen3.5 model; contributions for additional model templates are actively sought. The claude_code adapter is reserved, indicating potential future development or integration points.