NVIDIA ChatRTX is a developer reference project for building Retrieval Augmented Generation (RAG) chatbots on Windows. It enables users to create personalized chatbots that can query their own documents, notes, and images, offering fast, secure, and contextually relevant answers via voice or text input. The project targets developers and power users looking to leverage local LLM inference with NVIDIA hardware.

How It Works

ChatRTX utilizes a RAG pipeline that integrates TensorRT-LLM for optimized LLM inference, NVIDIA NIM microservices, and LlamaIndex for data indexing and retrieval. It employs the FAISS vector search library to efficiently find relevant information within a user's data. This architecture allows for local execution, enhancing privacy and speed by leveraging RTX GPU acceleration.

Quick Start & Requirements

• Installation: Download and run the installer from the project's release page.
• Prerequisites:
  • NVIDIA GeForce RTX 5090/5080, RTX 6000 Ada, or RTX 30/40 series GPU with at least 8GB VRAM.
  • Windows 11 23H2 or 24H2.
  • NVIDIA Driver 572.16 or later.
• Setup: The application downloads and installs additional third-party software. Review their licenses.
• Links: ChatRTX_APIs, ChatRTX_App

Highlighted Details

• Supports various file formats including text, PDF, DOC/DOCX, XML, PNG, JPG, BMP.
• Integrates multiple AI models like Llama 3.1 8B, Mistral 7B, Gemma 7B, and Whisper for voice input.
• Offers voice query capabilities through RIVA Parakeet and Whisper models.
• Provides ChatRTX APIs for integrating RAG features into custom applications.

Maintenance & Community

This is an official NVIDIA project, indicating strong backing and potential for ongoing development. Community channels or roadmaps are not explicitly mentioned in the README.

Licensing & Compatibility

The project itself appears to be a reference implementation. However, it relies on and installs third-party open-source software, whose licenses must be reviewed for compatibility, especially for commercial use.

Limitations & Caveats

The project is Windows-specific and requires a compatible NVIDIA RTX GPU with a minimum of 8GB VRAM. Users must review the licenses of all included third-party components.