JARVIS is an open-source system designed to connect Large Language Models (LLMs) with a wide array of AI models from the Hugging Face Hub, enabling complex AI task automation. It targets researchers and developers aiming to build sophisticated AI agents capable of planning, selecting, and executing diverse AI tasks. The system facilitates advanced AI research and provides a framework for integrating LLMs with specialized models for tasks like image generation, text analysis, and more.

How It Works

JARVIS employs a four-stage workflow: Task Planning (LLM disassembles user requests), Model Selection (LLM chooses appropriate Hugging Face models), Task Execution (system invokes selected models), and Response Generation (LLM synthesizes results). This approach leverages the LLM's understanding and planning capabilities to orchestrate specialized AI models, offering a flexible and powerful way to tackle complex, multi-modal AI problems.

Quick Start & Requirements

• Installation: Clone the repository and set up a Python 3.8 environment. Install PyTorch with CUDA 11.7 support. Run pip install -r requirements.txt.
• Prerequisites: OpenAI API Key, Hugging Face Token. For local model execution, significant VRAM (>= 24GB recommended) and disk space (>= 284GB for default config) are required. A "lite" configuration is available for using Hugging Face Inference Endpoints without local model deployment.
• Resources: The default configuration requires substantial GPU VRAM and disk space. The lite configuration is much lighter.
• Docs: HuggingGPT Paper, EasyTool Paper, TaskBench Paper.

Highlighted Details

• Supports OpenAI's Azure platform and GPT-4.
• Offers CLI, Server (with Web API), and Gradio demo modes.
• Includes experimental Docker support for NVIDIA Jetson devices.
• Recent releases include Easytool for enhanced tool usage and TaskBench for LLM evaluation.

Maintenance & Community

The project is actively developed by Microsoft. Recent updates in late 2023 and early 2024 indicate ongoing development and release of new components like Easytool and TaskBench.

Licensing & Compatibility

The repository does not explicitly state a license in the README. Compatibility for commercial use or closed-source linking is not specified.

Limitations & Caveats

The "lite" configuration relies on Hugging Face Inference Endpoints, which may have stability limitations. The default configuration has very high hardware requirements. The project was undergoing planning and rebuilding as of July 2023, with a new version expected.