This project provides a real-time, voice-driven conversational AI experience, enabling users to speak naturally with an LLM and receive spoken responses. It's designed for users seeking a fluid, interactive AI companion, offering low-latency communication through a client-server architecture.

How It Works

The system captures voice via the browser, streams audio chunks to a Python backend using WebSockets, and transcribes speech to text using RealtimeSTT. The text is processed by an LLM (Ollama or OpenAI), and the AI's response is synthesized into speech by RealtimeTTS. Audio is streamed back to the browser for playback, with support for interruptions and dynamic silence detection for natural turn-taking.

Quick Start & Requirements

• Docker Installation (Recommended):
  1. docker compose build
  2. docker compose up -d
  3. docker compose exec ollama ollama pull hf.co/bartowski/huihui-ai_Mistral-Small-24B-Instruct-2501-abliterated-GGUF:Q4_K_M
  • Installation & Setup
• Prerequisites:
  • NVIDIA GPU with CUDA 12.1 (highly recommended for performance).
  • Docker Engine and Docker Compose v2+ (for Docker setup).
  • Python 3.9+ (for manual setup).
  • Linux recommended for optimal Docker GPU integration.
  • PyTorch installation instructions for specific CUDA versions.

Highlighted Details

• Low-latency focus via audio chunk streaming and WebSockets.
• Pluggable LLM backends (Ollama default, OpenAI support).
• Customizable Text-to-Speech engines (Kokoro, Coqui, Orpheus).
• Web Interface using Vanilla JS and Web Audio API.

Maintenance & Community

• Contributions are welcome via issues and pull requests.

Licensing & Compatibility

• Core codebase: MIT License.
• External TTS engines and LLM providers have their own licenses; users must comply.

Limitations & Caveats

CPU-only or weaker GPU performance will be significantly slower. Manual installation, especially on non-Windows systems or with different CUDA versions, may require troubleshooting.