Visual Med-Alpaca is a parameter-efficient, multimodal foundation model tailored for the biomedical domain, built upon LLaMA-7B. It addresses the high computational costs of training large language models for specialized fields by integrating plug-and-play visual modules, enabling tasks like radiological image interpretation and clinical question answering. The target audience includes researchers and developers working on biomedical AI applications.

How It Works

Visual Med-Alpaca bridges text and vision through a prompt augmentation method. Medical images are processed by specialized "visual experts" (e.g., Med-GIT for radiology, DePlot for charts) which convert visual information into intermediate text formats. This text is then merged with textual queries by a prompt manager, which feeds into the Med-Alpaca LLM for generating domain-specific responses. This modular approach allows for cost-effective integration of diverse visual capabilities.

Quick Start & Requirements

• Install/Run: Gradio example code is provided for local dashboard creation.
• Prerequisites: Requires a single consumer GPU (e.g., NVIDIA GeForce RTX 3090 Ti for LoRA fine-tuning).
• Resources: LoRA fine-tuning took 6.55 hours on a single RTX 3090 Ti.
• Links: Data, Code, Models

Highlighted Details

• Trained on a 54k instruction dataset curated by GPT-3.5-Turbo and human experts.
• Integrates with specialized visual models like Med-GIT (radiology captioning) and DePlot (chart interpretation).
• Achieves parameter efficiency through techniques like LoRA.
• Demonstrates capabilities in interpreting radiological images and answering clinical questions.

Maintenance & Community

The project originates from the Language Technology Lab at the University of Cambridge. Further details on community engagement or ongoing development are not explicitly detailed in the README.

Licensing & Compatibility

The model inherits restrictions from LLaMA. Commercial or clinical use is strictly prohibited. It is intended for academic research purposes only.

Limitations & Caveats

Visual Med-Alpaca is strictly for academic research and not approved for any clinical use. Users are cautioned about potential inaccuracies or misleading medical advice, and reliance for medical decision-making is at the user's own risk.