VCoder enhances Multimodal Large Language Models (MLLMs) by integrating auxiliary perception modalities like segmentation and depth maps. It targets researchers and developers seeking improved object-level perception and control within MLLMs, offering a versatile adapter approach.

How It Works

VCoder acts as an adapter layer for MLLMs, enabling them to process additional visual cues like semantic segmentation and depth maps. This approach provides finer-grained object-level understanding, facilitating more precise interactions and reasoning in multimodal tasks without altering the core LLM architecture.

Quick Start & Requirements

Requires Python 3.10, PyTorch 2.0.1, and CUDA 11.7. Installation involves cloning the repo, setting up a Conda environment, installing specific PyTorch/CUDA builds, and pip install -e .. Dependencies include ninja and flash-attn. Pre-trained checkpoints for VCoder LLaVA-1.5 and VCoder-DS LLaVA-1.5 are available on HuggingFace Hub. A demo is accessible via HuggingFace Spaces, with CLI inference examples provided.

Highlighted Details

• Achieves strong performance on the COST dataset for object-level perception tasks (segmentation, depth estimation).
• VCoder LLaVA-1.5-13b models show competitive COST dataset benchmark results.
• Maintains base LLaVA-1.5 performance on general QA benchmarks as no parameters are finetuned for those tasks.
• Introduces the COST dataset for training and evaluating MLLMs on object-level perception.

Maintenance & Community

Associated with SHI-Labs (Jitesh Jain, Jianwei Yang, Humphrey Shi). Acknowledgements include LLaVA, OneFormer, DINOv2. No specific community channels (Discord, Slack) or roadmap details are provided, though a future version is indicated as "COMING SOON".

Licensing & Compatibility

The README does not specify a software license. This omission may pose compatibility concerns for commercial use or integration into closed-source projects.

Limitations & Caveats

Performance on general multimodal QA benchmarks mirrors the base LLaVA-1.5 model, as VCoder focuses on object-level perception enhancement via auxiliary inputs. Specific hardware and software versions (Python 3.10, PyTorch 2.0.1, CUDA 11.7) are required for setup. Future developments are indicated as "COMING SOON".