This project provides a highly customizable, web-based framework for creating AI-powered virtual companions, targeting developers and users interested in interactive digital humans. It enables real-time voice and text interaction, visual character display, and advanced AI capabilities, offering a flexible platform for personalized AI experiences.

How It Works

The framework integrates various AI models for speech recognition (SenseVoice), language understanding (LLMs compatible with OpenAI API, DeepSeek-R1), and speech synthesis. It supports multimodal input via screen capture, camera, or uploaded images, and can leverage local knowledge bases like AnythingLLM and Dify for enhanced context. A key feature is the optional Letta framework for long-term memory, addressing the stateless nature of traditional LLMs by managing state across sessions.

Quick Start & Requirements

• Install: Clone the repository (git clone https://github.com/swordswind/ai_virtual_mate_web.git), install dependencies (pip install -r requirements.txt, pip install flet==0.13.0, pip install letta==0.6.7), download SenseVoice models to data/model, and run python main.py.
• Prerequisites: Windows 10+, Intel Core i5 8th/AMD R5 3000+, 8GB RAM, Intel UHD 620/AMD Vega 7 graphics, 2GB storage, microphone, 720P camera.
• Setup: Requires downloading speech models and potentially configuring API keys in data/set/key_set.txt.
• Docs: https://github.com/swordswind/ai_virtual_mate_web

Highlighted Details

• Supports real-time voice interaction with interruptible speech synthesis.
• Features Live2D and MMD 3D character rendering with interactive elements.
• Offers local knowledge base integration for improved AI responses.
• Accessible across devices on a local network via web browser.

Maintenance & Community

The project is actively maintained by swordswind. Contact is available via email (swordswind@qq.com) and GitHub.

Licensing & Compatibility

• License: GPL-3.0.
• Compatibility: GPL-3.0 is a strong copyleft license, requiring derivative works to also be open-sourced under GPL-3.0. This may restrict integration into closed-source commercial products without careful consideration.

Limitations & Caveats

The project is primarily designed for Windows. While it supports various AI models, performance and compatibility may vary. Some features like real-time voice and camera input on non-host devices require additional software (e.g., AudioReplay, iVCam). Users may encounter false positives from antivirus software.