This project provides a multilingual AI companion that interacts vocally and visually through VTube Studio, aiming to be a therapeutic and responsive agent. It's designed for users interested in virtual companions, smart home integration, and immersive AI experiences.

How It Works

The system processes user speech via SpeechRecognition, transcribes it using OpenAI's Whisper, and stores conversations in a vector database (hyperdb) for context retrieval. Responses are generated by a local LLM (llama.cpp) and spoken using VITS TTS, with logic for app control and smart home integration. This approach leverages local models for privacy and responsiveness, while integrating with cloud services for advanced speech-to-text.

Quick Start & Requirements

• Install: Clone the repository, run python setup.py, edit .env, and execute OneReality.bat.
• Prerequisites: Python 3.10.11, GIT, CUDA 11.7 (Nvidia GPU), Visual Studio Community 2022 (C++ dev), VTube Studio, VB Cable Audio Driver.
• Setup: Requires manual audio routing configuration via Sound Mixer Options and VTube Studio settings. Smart home control requires a Tuya cloud project.
• Links: Demo Video, Discord

Highlighted Details

• Supports English, Japanese, Korean, and Chinese.
• Features short-term and long-term memory.
• Integrates with smart home devices via Tuya Cloud IoT.
• Supports VR, hologram box, or desktop display via VTube Studio.

Maintenance & Community

The project is actively under development with significant updates planned, including a compiled executable. A Discord server is available for community support and discussion.

Licensing & Compatibility

Distributed under the GNU General Public License v3.0 (GPL-3.0). This license is copyleft and may have implications for commercial or closed-source use.

Limitations & Caveats

The project is explicitly stated to be in a state of major changes, with potential for errors during installation. The README warns against attempting installation "right now." Lip-sync functionality may require a system restart.