UniRL is a reinforcement learning framework designed for unified multimodal models, enabling a single RL post-training loop across diverse architectures like diffusion, autoregressive, and unified models. It targets researchers and engineers seeking to streamline RL fine-tuning for multimodal AI, offering a composable system to enhance model performance and flexibility.

How It Works

The framework centralizes multimodal RL training via a consistent post-training loop: sample generation, scoring, advantage calculation, policy update, and weight synchronization. UniRL employs a layered, composable architecture where Hydra configuration files define model, algorithm, and runtime specifics. Domain-specific trainers (e.g., DiffusionTrainer, ARTrainer) orchestrate pluggable components—rollout engines, algorithms, model bundles, reward services—and integrate with distributed runtimes like Ray DevicePool, FSDP, and Transfer Queue for efficient, scalable training.

Quick Start & Requirements

• Installation: Follow instructions in INSTALL.md.
• Primary Run Command: python -m unirl.train_diffusion --config-name=diffusion/sd3_trainside --cfg job --resolve or bash examples/run_experiment_single_node.sh diffusion/sd3_trainside.
• Prerequisites: Python 3.12+.
• Documentation: https://unirl-project.github.io/unirl/
• Community: WeChat Group

Highlighted Details

• Team-Proposed Algorithms: Features DRPO ("Rethinking the Divergence Regularization in LLM RL") and Flow-DPPO ("Flow-DPPO: Divergence Proximal Policy Optimization for Flow Matching Models"), each with tutorials.
• Broad Model Support: Integrates with numerous models including Stable Diffusion 3/3.5, Qwen-Image, FLUX.2-Klein, WAN 2.1/2.2, HunyuanVideo 1.0/1.5, Qwen-VL, Qwen3, Prompt-Enhancer, HunyuanImage3, and Bagel.
• Unified Training Modes: Supports distinct training domains: diffusion, autoregressive (AR), prompt-enhancer (PE), and unified models, each with example configurations.

Maintenance & Community

The project is actively maintained with a roadmap focused on expanding model and algorithm coverage, particularly for newer families like FLUX.2-Klein and HunyuanVideo. Contributions are welcomed via issues and pull requests following repository conventions. A WeChat group is available for community interaction.

Licensing & Compatibility

• License: Apache-2.0.
• Compatibility: The Apache-2.0 license is generally permissive for commercial use and integration with closed-source projects.

Limitations & Caveats

The README does not explicitly detail limitations. The roadmap indicates ongoing development, suggesting that support for certain models or algorithms may be under active expansion. Users should consult the roadmap and issue tracker for the latest coverage and potential gaps.