ChartLlama: A Multimodal LLM for Chart Understanding and Generation

ChartLlama is a multimodal LLM focused on chart understanding and generation. It addresses the need for automated chart interpretation, manipulation, and creation, targeting researchers and developers in data visualization and analysis. The model offers capabilities to redraw charts from visual input, edit them via instructions, and generate new charts from raw data, aiming to streamline complex data visualization workflows.

How It Works

The core approach involves training a multimodal LLM on a custom-generated instruction-tuning dataset. This enables ChartLlama to process visual chart inputs alongside textual commands, facilitating sophisticated chart manipulation. Key functionalities include redrawing charts based on visual examples, editing charts according to specific instructions, and generating novel charts from raw data, leveraging instruction tuning for precise control.

Quick Start & Requirements

Installation is straightforward via pip install -e .. A critical prerequisite is the prior setup of LLaVA-1.5. Inference relies on LLaVA's model_vqa_lora module and requires specific command-line configurations detailed in the repository. Users should anticipate managing LLaVA-1.5 dependencies and potential CUDA requirements for GPU acceleration.

Highlighted Details

• Multimodal LLM for chart understanding and generation.
• Chart redrawing from visual input.
• Instruction-based chart editing.
• New chart generation from raw data.

Maintenance & Community

Developed by researchers from Tencent and Nanyang Technological University. Specific community channels or active maintenance indicators are not detailed in the provided README.

Licensing & Compatibility

Licensed strictly for "RESEARCH purposes" and limited to "personal/research/non-commercial purposes." This restriction prohibits commercial use and integration into proprietary systems.

Limitations & Caveats

Training scripts and the full dataset are not yet open-sourced, impacting full reproducibility and community-driven development. The project's reliance on LLaVA-1.5 may introduce version dependencies and potential integration complexities.