Summary

AvatarAI is a production-ready, open-source platform for creating real-time AI avatar conversations. It enables users to upload a photo, clone a voice from a short audio clip, and engage in interactive, lip-synced video dialogues. This self-hostable solution targets engineers and power users, offering advanced features like barge-in and local-first operation for deployable AI avatar systems.

How It Works

The system employs a token-streaming pipeline for fluid interaction. User speech is processed by Whisper STT, sent to an LLM (Claude, GPT, Ollama), and the LLM's response is streamed token-by-token. Chatterbox TTS generates audio, and MuseTalk creates per-sentence lip-synced video chunks, streamed to the client. This allows video playback to begin before the LLM finishes, supporting real-time interruption ("barge-in") for natural dialogue.

Quick Start & Requirements

• Primary Install: Docker Compose for development (docker compose up -d). Manual installation supported.
• Prerequisites: Docker, Python 3.10+, Node.js 18+, FFmpeg, PostgreSQL, Redis. A CUDA-enabled NVIDIA GPU is highly recommended for real-time MuseTalk lip-sync; CPU performance is significantly degraded. LLM API keys (Anthropic, OpenAI) needed for cloud LLMs.
• Resource Footprint: MuseTalk models require ~9 GB download. GPU VRAM (16GB+) crucial for performance.
• Links: Swagger Docs: http://localhost:8000/docs. AWS deployment script: scripts/deploy-aws.sh.

Highlighted Details

• Zero-Shot Voice Cloning: Chatterbox Multilingual supports cloning voices from 10s audio clips across 23 languages.
• Real-Time Interaction: Features WebSocket streaming, token-by-token LLM output, and "barge-in" for immediate interruption.
• Local-First Operation: Supports running LLMs locally via Ollama, local Whisper, and local storage for privacy.
• Production-Grade: Includes JWT auth, rate limiting, persistent storage (Postgres, S3), Prometheus, CI, and tests.
• AWS Deployment: One-command script for EC2 GPU instances (g5.xlarge recommended for ~30 FPS).

Maintenance & Community

A public roadmap outlines future enhancements like WebRTC streaming and embeddable widgets. Contributions are welcomed. Prebuilt Docker images are available on GHCR.

Licensing & Compatibility

Licensed under MIT, permitting commercial use and integration into closed-source projects.

Limitations & Caveats

MuseTalk lip-sync performance is drastically reduced on CPU, making real-time interaction impractical without a GPU. Initial model loading for MuseTalk can introduce a startup delay. Features like hands-free mode are still under development.