This project enables real-time, voice-activated conversations with NPCs in any game, enhancing immersion by allowing players to speak directly to characters. It targets players of open-world games like Cyberpunk 2077 and Assassin's Creed, offering a novel way to interact with existing game worlds without modifying game code.

How It Works

The system captures microphone input, transcribes speech, and uses facial recognition to identify the target NPC. This information, along with character-specific data from vector stores and pre-conversation files, is fed to an LLM for response generation. The LLM's output is converted to speech and used to create lip-synced facial animations via SadTalker, which then replaces the NPC's in-game face pixels. Webcam input can also be used for emotion recognition to tailor NPC responses.

Quick Start & Requirements

• Installation: Clone repo, create/activate Python 3.10.6 virtual environment, pip install -r requirements.txt, run scripts/download_models.sh, then webui.bat from the sadtalker directory.
• Prerequisites: Python 3.10.6, Git, wget, Microsoft Build Tools/Visual Studio (Windows), FFmpeg.
• API Keys: Requires a Cohere API key (free trial available). GPT-4 integration is optional.
• Setup: Model downloads and initial setup may take time. Detailed setup instructions and Jupyter notebooks are provided for game-specific configuration.
• Links: Demo: https://twitter.com/Akshit2089/status/1673687342438051847, Explanation: https://twitter.com/cohere/status/1687131527174672384, Tutorial: https://youtu.be/6SHTlKYKCbs, Discord: https://discord.gg/CfK7DCWKwy

Highlighted Details

• Integrates LLM-powered dialogue and facial animation (SadTalker) into live gameplay.
• Utilizes vector stores for persistent NPC memory and pre-conversation files for dialogue style.
• Supports emotion recognition via webcam for more dynamic NPC interactions.
• Allows non-visual interaction by prefixing dialogue with the NPC's name.

Maintenance & Community

• Active development with a Discord server for community support and contributions.
• Encourages community contributions for expanding game compatibility and improving features.

Licensing & Compatibility

• The repository itself appears to be under a permissive license, but the underlying dependencies like SadTalker may have their own licenses. Specific licensing details for the combined project are not explicitly stated in the README.

Limitations & Caveats

The project relies on specific versions of Python and external tools, and setup can be complex, especially for non-Windows users or those unfamiliar with LLM integration and multimedia processing. Performance may vary based on hardware, particularly for real-time facial animation rendering.