Chameleon is a framework for compositional reasoning that augments Large Language Models (LLMs) with various tools, enabling them to synthesize programs for complex tasks. It is designed for researchers and practitioners looking to enhance LLM capabilities in areas like question answering and data analysis.

How It Works

Chameleon utilizes an LLM as a natural language planner to infer and compose a sequence of tools. These tools can include other LLMs, vision models, search engines, Python functions, and rule-based modules. This approach allows for flexible and adaptive problem-solving by dynamically combining specialized functionalities to address diverse user queries.

Quick Start & Requirements

• Installation: pip install -r requirements.txt
• Prerequisites: Python 3.8.10, OpenAI API key (paid account required), optional Bing Search API key.
• Setup: Requires obtaining API keys and potentially downloading datasets.
• Documentation: Project Page, Paper

Highlighted Details

• Achieves 86.54% accuracy on ScienceQA (11.37% improvement over prior few-shot models) and 98.78% on TabMWP (17.0% increase over SOTA) using GPT-4.
• Demonstrates adaptability by composing tools like search engines for domain-specific knowledge and Python for precise computations.
• Supports visualization of predictions and module transition graphs via provided Jupyter Notebooks.
• Offers a structured approach for developing new tasks by defining module inventories and LLM planners.

Maintenance & Community

The project is associated with Pan Lu and has received recognition from various AI publications and platforms. Contact information and discussion channels are available via the project page and Twitter.

Licensing & Compatibility

The repository is released under a permissive license, suitable for commercial use and integration with closed-source applications.

Limitations & Caveats

The README mentions that live calling for image captioner and text detector modules is "coming soon." Some performance gains on ScienceQA are dependent on the optional Bing Search API key.