Summary

ChartQA introduces a benchmark dataset and associated models for question answering tasks involving charts, demanding both visual and logical reasoning. It targets researchers and engineers developing AI systems for chart comprehension, offering a challenging dataset and advanced models like UniChart, ChartInstruct, and ChartGemma to advance capabilities in multimodal understanding and data analysis from visual representations.

How It Works

The project leverages multimodal approaches, integrating visual chart elements with textual queries. Models such as VL-T5, T5, and VisionTapas are employed to process chart images, their underlying data tables (CSV), and detailed annotations (e.g., bounding boxes for bars, slices, text). This enables sophisticated reasoning over visual and structured information to answer complex questions about chart content.

Quick Start & Requirements

• Installation: Models are described as user-friendly and runnable with minimal code. Public web demos are available. Specific installation commands are not detailed in the provided text.
• Dataset: The ChartQA dataset, including a full version with annotations, is available within the repository and via a Hugging Face dataset link.
• Prerequisites: Not explicitly detailed, but likely requires a Python environment and deep learning libraries. GPU acceleration is recommended for model performance.
• Links:
  • Paper: ChartQA (https://aclanthology.org/2022.findings-acl.177)
  • Full Dataset: Available on Hugging Face.

Highlighted Details

• ChartQAPro: A new, more diverse, and challenging dataset for real-world chart question answering.
• UniChart: A 140M parameter model proficient in ChartQA, Chart-to-Table, Chart Summarization, and Open-ended QA.
• ChartInstruct: An LLaVA-based Chart LLM supporting Llama2 (7B) and Flan-T5-XL (3B).
• ChartGemma: A state-of-the-art Chart LLM built on PaliGemma (3B) for visual reasoning.
• Detailed Annotations: The dataset includes comprehensive annotations for bar, line, and pie charts, detailing bounding boxes, labels, axes, and figure elements.

Maintenance & Community

• Contact: Primary contact is Ahmed Masry (amasry17@ku.edu.tr).
• Community/Roadmap: No specific community channels (e.g., Discord, Slack) or roadmap details are provided in the excerpt.

Licensing & Compatibility

• License: The license type is not specified in the provided README excerpt.
• Compatibility: No specific compatibility notes for commercial or closed-source use are mentioned.

Limitations & Caveats

• Annotation Quality: Dataset annotations are noted as potentially noisy due to automated SVG processing and manual annotation efforts for certain charts.
• Data Availability: Some chart images lacked source SVG files, necessitating heuristic-based manual annotation.
• Dataset Versions: Multiple versions of the dataset exist, with varying levels of annotation detail.