GUI-R1: A Generalist R1-Style Vision-Language Action Model For GUI Agents

GUI-R1 addresses the challenge of building generalist vision-language action models for GUI agents. It enables Large Vision-Language Models (LVLMs) to execute tasks across diverse platforms (Windows, Linux, MacOS, Android, Web) with significantly less training data than prior state-of-the-art methods. The project targets researchers and developers aiming to enhance GUI automation and agent capabilities by leveraging reinforcement learning for improved execution.

How It Works

GUI-R1 employs a reinforcement learning approach, specifically Group Relative Policy Optimization (GRPO), to train its policy model. Given high-level instructions, action history, and visual inputs, the model generates reasoning steps and actions. Training is driven by verifiable rewards derived from action type, click point, and input text, optimizing a unified action space rule model. This method allows for superior performance using a fraction of the data required by previous large-scale pretraining approaches.

Quick Start & Requirements

The project recommends using pre-built Docker images for setup.

• Primary install/run command: Docker pull hiyuga/verl:ngc-th2.5.1-cu120-vllm0.7.4-hotfix (stable) or hiyuga/verl:ngc-th2.6.0-cu120-vllm0.8.2 (nightly).
• Prerequisites: CUDA >= 12.0, vLLM versions 0.7.4 or 0.8.2.
• Data: Download the GUI-R1-3K dataset, structured as specified in the README.
• Links: Paper: https://arxiv.org/abs/2504.10458.

Highlighted Details

• Achieves superior performance across eight benchmarks on mobile, desktop, and web platforms using only 0.02% of the data (3K samples) compared to methods like OS-Atlas (13M samples).
• Leverages a small, carefully curated dataset across multiple operating systems (Windows, Linux, MacOS, Android, Web).
• Released an 800K high-quality reinforcement learning dataset filtered from OS-Atlas using QwenVL2.5-7B, with a diverse 10K subset optimized using DAPO.

Maintenance & Community

The project acknowledges contributions from DeepSeek, VLM-R1, QwenVL, EasyR1, and OS-ATLAS. No specific community links (Discord, Slack) or roadmap details are provided in the README.

Licensing & Compatibility

The README does not explicitly state the license type or provide compatibility notes for commercial use.

Limitations & Caveats

The README does not detail specific limitations, alpha status, known bugs, or unsupported platforms. The project appears to be recently released with weights and code available as of April 2025.