This project provides "Omega," a napari-aware, autonomous LLM-based agent for image processing and analysis. It targets researchers and users who want to interact with image analysis tools conversationally, leveraging OpenAI's ChatGPT API and LangChain to automate tasks, generate code, and create napari widgets.

How It Works

Omega acts as an LLM-powered agent that can converse with users, execute image processing tasks, write and debug Python code for napari, and even generate custom napari widgets. It utilizes OpenAI's ChatGPT API, integrated via the LangChain library, to understand user requests and employ various tools to manipulate images and control the napari viewer. This approach allows for dynamic, conversational image analysis and workflow automation.

Quick Start & Requirements

• Install via pip: pip install napari-chatgpt or through napari's plugin installer.
• Requires an OpenAI API key. GPT-4 is recommended for best results.
• Detailed setup and usage instructions are available on the project's wiki.

Highlighted Details

• Demonstrates autonomous code generation, debugging, and napari widget creation.
• Achieves a reported ~90% success rate in tasks with GPT-4, as detailed in a preprint.
• Features an AI-Augmented Code Editor with syntax highlighting, code completion, and network code sharing.
• Can be used for learning image analysis workflows and sharing custom tools.

Maintenance & Community

The project was initiated as a weekend project by Loic A. Royer, who leads a research group at the Chan Zuckerberg Biohub. Contributions are welcome, with tests runnable via tox.

Licensing & Compatibility

Distributed under the BSD-3 license, allowing for commercial use and integration with closed-source projects.

Limitations & Caveats

Requires an OpenAI API key, limiting its use to those with access and incurring potential costs. The project includes a disclaimer warning about the agent's potential to misunderstand requests or execute unintended actions, recommending sandboxing for potentially problematic experiments.