Robust-U1 addresses the challenge of multimodal large language models (MLLMs) failing to understand corrupted visual content. It introduces a unified MLLM capable of self-recovering degraded images and reasoning over them, enabling robust visual understanding in real-world scenarios. This project is targeted at researchers and engineers seeking to improve MLLM resilience to visual noise and artifacts.

How It Works

The core innovation lies in a three-stage training pipeline. Initially, a base MLLM (BAGEL-7B-MoT) undergoes supervised fine-tuning (SFT) using the MathCanvas framework for visual self-recovery. This is followed by reinforcement learning (RL) with Flow-GRPO to align the recovery process with pixel-level fidelity and semantic consistency. Finally, the model is trained for multimodal reasoning over both corrupted and recovered images, again leveraging MathCanvas. This approach allows MLLMs to explicitly model and correct visual degradations internally, surpassing limitations of black-box alignment or text-only compensation.

Quick Start & Requirements

Installation involves cloning the repository (https://github.com/jqtangust/Robust-U1.git), creating a Python 3.10 Conda environment, and installing dependencies via requirements.txt and pip install -e .. Users need to download the base BAGEL-7B-MoT model (https://huggingface.co/ByteDance-Seed/BAGEL-7B-MoT) and the Robust-U1 checkpoints (https://huggingface.co/Jiaqi-hkust/Robust-U1), available on Hugging Face. An online demo is provided via Hugging Face Spaces (https://huggingface.co/spaces/Jiaqi-hkust/Robust-U1). Training requires cloning and setting up external projects like MathCanvas (https://github.com/shiwk24/MathCanvas.git) and Flow-GRPO (https://github.com/yifan123/flow_grpo.git), along with preparing specific datasets.

Highlighted Details

• Official implementation for the ICML 2026 paper "Robust-U1: Can MLLMs Self-Recover Corrupted Visual Content for Robust Understanding?".
• Offers pretrained models and a functional online demo on Hugging Face.
• Features a novel three-stage training methodology integrating SFT and RL for visual self-correction.
• Performance is benchmarked using VLMEvalKit (https://github.com/open-compass/VLMEvalKit.git) for anti-degradation capabilities and R-Bench (https://github.com/Q-Future/R-Bench.git) for real-world corruption resilience.

Maintenance & Community

The project is associated with several academic authors, with Jiaqi Tang listed as a primary contact (jtang092@connect.ust.hk). No specific community channels (e.g., Discord, Slack) are detailed. The project acknowledges dependencies on and contributions from the BAGEL, MathCanvas, and Flow-GRPO open-source communities.

Licensing & Compatibility

The project is released under the permissive MIT License, which generally allows for commercial use and integration into closed-source projects.

Limitations & Caveats

As an official implementation for a recent conference paper (ICML 2026), the project may still be under active development. Detailed hardware requirements for training and inference are not specified in the README. Setting up the full training pipeline necessitates integrating and configuring several external codebases and datasets.