BLIVA is a multimodal large language model designed to excel at visual question answering, particularly for images containing significant amounts of text. It targets researchers and developers working with visual understanding tasks, offering improved performance on text-rich benchmarks.

How It Works

BLIVA builds upon the BLIP-2 architecture, integrating a visual encoder with a language model. Its novelty lies in its specific training methodology and data augmentation strategies tailored for text-heavy visual question answering. This approach allows BLIVA to effectively extract and interpret textual information embedded within images, leading to more accurate responses compared to general-purpose multimodal models.

Quick Start & Requirements

• Install: pip install -e . after cloning the repository.
• Prerequisites: Python 3.9, PyTorch. Vicuna version requires downloading Vicuna-7B weights and specifying paths. FlanT5 version automatically downloads weights.
• Resources: Training requires 8x A6000 Ada GPUs. Inference is less demanding.
• Links: Demo, Vicuna Weights, FlanT5 Weights, Paper.

Highlighted Details

• Achieved No.1 in Color, Poster, and Commonsense Reasoning subtasks on the MME benchmark.
• Outperforms strong baselines like InstructBLIP and LLaVA on text-rich VQA benchmarks.
• Offers both Vicuna-7B and FlanT5-XXL versions.
• Released a new dataset, YTTB-VQA, for text-rich visual question answering.

Maintenance & Community

The project is associated with UC San Diego and Coinbase. Key updates are announced via GitHub releases.

Licensing & Compatibility

• Code: BSD 3-Clause License.
• Vicuna Model Weights: Subject to LLaMA's model license.
• FlanT5 Model Weights: Apache 2.0 License (available for commercial use).
• YTTB-VQA Data: CC BY NC 4.0.

Limitations & Caveats

The Vicuna version's commercial use is restricted by LLaMA's license. While strong on text-rich VQA, performance on general VQA benchmarks may vary.