This project provides a real-time conversational AI system capable of cloning voices and personalities, addressing the fragmentation in existing voice AI solutions. It targets users seeking immersive, personalized AI interactions, enabling conversations with AI clones that mimic specific individuals' voices and speaking styles with low latency, akin to a phone call.

How It Works

The system integrates Speech-to-Text (STT), a Large Language Model (LLM) with personality and memory modules, and Text-to-Speech (TTS) with voice cloning. User audio is transcribed, processed by the LLM for context-aware responses incorporating personality traits, and then synthesized into speech using a cloned voice. This end-to-end pipeline is engineered for sub-second latency, offering a seamless conversational experience.

Quick Start & Requirements

• Primary install/run command: git clone https://github.com/YeJe-cpu/talk-to-fengge.git, cd talk-to-fengge, then use an AI coding assistant or follow manual setup.
• Non-default prerequisites: Python 3.12+, uv (recommended) or pip, and a self-hosted LiveKit Server.
• Dependencies: API keys for Cartesia (STT), MiniMax (LLM), and a chosen TTS provider. The recommended VoxCPM TTS requires an NVIDIA GPU with >= 8GB VRAM. MOSS-TTS is a CPU-fallback option.
• Links: LiveKit Docs, Cartesia, MiniMax.

Highlighted Details

• Real-time voice conversations with AI clones, simulating phone calls.
• Voice cloning from brief audio samples (15-45 seconds).
• Personality injection, replicating speaking style, catchphrases, and thought patterns.
• Engineered latency under 1 second, with actual perceived latency around 2-3 seconds.
• Modular architecture supporting various STT, LLM, and TTS providers.

Maintenance & Community

Contributions via Fork + PR are welcomed. Users can open Issues for questions, suggestions, or feature requests. Contact is available via X/Twitter. Future development plans are outlined in the README.

Licensing & Compatibility

The project is released under the MIT license, permitting commercial use and integration into closed-source projects.

Limitations & Caveats

Deployment involves managing multiple API keys and services, lacking a one-click solution. Actual conversational latency is higher than engineering latency due to network and API response times. The memory system is primarily event-driven and may be less effective when the AI dominates the conversation. The current frontend is a basic single-page application without advanced avatars.