RLAIF-V is an open-source framework for aligning multimodal large language models (MLLMs) to achieve enhanced trustworthiness, specifically targeting GPT-4V level performance. It is designed for researchers and developers working on improving MLLM reliability and reducing hallucinations.

How It Works

RLAIF-V leverages AI feedback, both in the form of a high-quality preference dataset and an online feedback learning algorithm. The approach focuses on generating and utilizing preference pairs to train models, aiming to reduce hallucinations and improve performance across generative and discriminative tasks. The framework supports iterative alignment, allowing for continuous improvement through cycles of data generation, training, and evaluation.

Quick Start & Requirements

• Install: Clone the repository, activate a conda environment (conda create -n rlaifv python=3.10, conda activate rlaifv), and install dependencies (pip install -e .). Install the en_core_web_trf-3.7.3 spaCy model.
• Prerequisites: Python 3.10, Conda, spaCy, Hugging Face datasets, COCO2014 dataset annotations for Object HalBench evaluation. OpenAI API key is required for evaluation metrics.
• Resources: Model weights are available for 7B and 12B parameter versions. Training requires significant computational resources.
• Links: Paper, Code, Weights

Highlighted Details

• Achieves "super GPT-4V trustworthiness" on generative and discriminative tasks with its 12B model.
• Features an open-source multimodal preference dataset with 83,132 comparison pairs.
• Demonstrates inference-time scaling benefits from RLAIF-V reward.
• Supports LoRA training for efficient fine-tuning.

Maintenance & Community

The project is associated with the RLHF-V organization. Key contributors are listed in the paper citations. Community engagement channels are not explicitly mentioned in the README.

Licensing & Compatibility

The data, code, and checkpoints are licensed for research use only. Usage is restricted by the licenses of LLaMA, Vicuna, and Chat GPT. The dataset is CC BY NC 4.0, prohibiting commercial use.

Limitations & Caveats

The project is primarily for research purposes and has non-commercial restrictions due to its dataset license. Evaluation for Object HalBench requires the COCO2014 dataset and an OpenAI API key.