Chatty provides a private, in-browser interface for interacting with large language models (LLMs), aiming to replicate the user experience of popular AI chat platforms. It targets users who prioritize data privacy and offline functionality, enabling them to run models like Gemma, Llama, and Mistral directly on their local hardware without server-side processing.

How It Works

Chatty leverages WebGPU for client-side LLM execution, allowing models to run directly within the browser. This approach ensures data privacy as all processing occurs locally. It utilizes Xenova's Transformer.js for model inference and LangChain.js for features like file-based Q&A and custom memory, enabling offline use and local document analysis.

Quick Start & Requirements

• Install: npm install and npm run dev for local development.
• Prerequisites: Node.js (18+), npm. Docker support is available but not production-optimized.
• Hardware: A GPU with sufficient VRAM is recommended for efficient model operation (e.g., 6GB for 7B models, 3GB for 3B models).
• Demo: https://chattyui.com/

Highlighted Details

• Runs LLMs (Gemma, Llama2/3, Mistral) locally via WebGPU.
• Supports chat with local files (PDF, text, code) using TransformerEmbeddings.
• Features chat history, custom memory, voice input, and markdown/code highlighting.
• Responsive design for desktop and mobile (WebGl support).

Maintenance & Community

Maintained by Addy Osmani and Jakob Hoeg Mørk. Contributions are welcome via contributing guidelines.

Licensing & Compatibility

The project appears to be MIT licensed, allowing for commercial use and integration with closed-source applications.

Limitations & Caveats

WebGPU support is primarily for Chrome and Edge, with experimental support in Firefox. The Dockerfile is not optimized for production. The roadmap indicates future support for multiple file embeddings and a prompt management system.