h2oGPT provides a private, offline platform for interacting with local Large Language Models (LLMs) and processing various document types. It targets users who need secure, self-hosted AI capabilities for tasks like document summarization, Q&A, and general chat, offering a comprehensive alternative to cloud-based services.

How It Works

h2oGPT leverages a flexible architecture supporting multiple LLM backends (e.g., Llama.cpp, Hugging Face) and embedding models for accurate document retrieval. It employs techniques like HYDE and Semantic Chunking for enhanced retrieval, Attention Sinks for extended context, and parallel processing for high throughput. The system supports both Gradio and CLI interfaces, with an OpenAI-compliant API for seamless integration.

Quick Start & Requirements

• Installation: Docker is recommended for full capabilities across Linux, Windows, and macOS. Linux scripts are also available.
• Prerequisites: GPU with CUDA is recommended for advanced features like Semantic Chunking and faster inference. CPU support is available.
• Resources: Detailed setup and running guides are available for various platforms and configurations.
• Demos: Live Gradio and OpenWebUI demos are provided.

Highlighted Details

• Supports a wide array of document types including PDFs, Word, Excel, images, video frames, audio, and code.
• Integrates vision models (LLaVa, Gemini-Pro-Vision) and image generation (Stable Diffusion).
• Features voice interaction with Whisper STT and Microsoft Speech T5 TTS, including voice cloning.
• Offers an OpenAI-compliant Server Proxy API, acting as a drop-in replacement for OpenAI's services.

Maintenance & Community

The project is actively developed by H2O.ai, a company with a strong background in enterprise AI and open-source ML platforms. Community support channels include Discord.

Licensing & Compatibility

Licensed under Apache 2.0, allowing for commercial use and integration with closed-source projects.

Limitations & Caveats

The README includes a disclaimer regarding potential biases and inaccuracies in LLM outputs, emphasizing user responsibility for evaluating generated content. Some advanced features may require specific hardware (e.g., GPU for Semantic Chunking).