LLaVA-RLHF introduces a novel approach to align Large Multimodal Models (LMMs) using Factually Augmented Reinforcement Learning from Human Feedback (Fact-RLHF). This method aims to improve visual reasoning and perception by augmenting the reward model with factual information like image captions, mitigating reward hacking and enhancing performance. It is targeted at researchers and developers working with LMMs who seek to improve their factual accuracy and alignment.

How It Works

The core innovation is Fact-RLHF, which enhances the standard RLHF process by incorporating explicit factual data into the reward model. This augmentation helps the model learn more robust and factually grounded responses, addressing common issues in RLHF where models might exploit loopholes in the reward function. The project leverages the LLaVA architecture, building upon its visual and language understanding capabilities.

Quick Start & Requirements

• Install/Run: Refer to the demo directory for inference instructions. Training pipelines are available in SFT and RLHF directories.
• Prerequisites: Training requires 8 x A100 GPUs with 80GB memory. For fewer GPUs, adjust per_device_train_batch_size and gradient_accumulation_steps while maintaining the global batch size.
• Resources: Training is resource-intensive.
• Links: Project Page, Demo, Model Weights

Highlighted Details

• First open-source RLHF-trained LMM for general visual and language understanding.
• Fact-RLHF algorithm designed to alleviate reward hacking.
• Achieves impressive visual reasoning and perception capabilities.

Maintenance & Community

The project cites contributions from various open-source efforts, including Meta LLaMA, Stanford Alpaca, Vicuna, LLaVA, QLoRA, Hugging Face PEFT, and AlpacaFarm. Specific community links (Discord/Slack) are not provided in the README.

Licensing & Compatibility

The README does not explicitly state a license for the LLaVA-RLHF code or models. It references LLaVA, which is typically under an Apache 2.0 license, but this should be verified for the RLHF components.

Limitations & Caveats

Training is highly resource-intensive, requiring multiple high-end GPUs. The specific licensing for the RLHF components needs explicit confirmation for commercial or closed-source integration.