LanguageLeapAI provides real-time AI-powered translation for cross-lingual voice communication, targeting users who wish to converse with others speaking different languages. It offers both voice translation and audio subtitling capabilities, enabling seamless interaction in online games or other communication platforms.

How It Works

The system comprises two Python scripts leveraging free, open-source AI models. voice_translator.py captures microphone input via push-to-talk, transcribes it using WhisperAI, translates the text with DeepL (or Google Translate as a fallback), and synthesizes speech in the target language using Voicevox (or Thorsten for German). subtitler.py captures application audio output, transcribes and translates speech to English using WhisperAI, and displays it as on-screen subtitles using Tkinter. The architecture relies on Docker Compose for deploying WhisperAI and Voicevox, with optional Google Colab integration for GPU-accelerated processing.

Quick Start & Requirements

• Installation: Primarily via Docker Compose (docker-compose up -d) for WhisperAI and Voicevox, followed by running Python scripts (python voice_translator.py or python subtitler.py) from the src/ directory.
• Prerequisites: Docker, Docker Compose, Python 3.x, DeepL API key (free tier available), microphone/speaker setup. Google Colab notebooks are provided for GPU acceleration.
• Setup: Requires installing services, configuring audio routing, and setting up an environment file.
• Links: Setup Guide: https://www.youtube.com/watch?v=bN5UaEkIPGM, Showcase: https://www.youtube.com/watch?v=UY7sRB60wZ4

Highlighted Details

• Integrates OpenAI's WhisperAI for ASR and translation, DeepL for text translation, and Voicevox (Japanese) or Thorsten (German) for TTS.
• Supports real-time voice chat translation and on-screen subtitling.
• Offers flexibility to use only the subtitler or voice translator components.
• Provides Google Colab notebooks for users with less powerful local hardware.

Maintenance & Community

The project is maintained by SociallyIneptWeeb. Community links or roadmaps are not explicitly mentioned in the README.

Licensing & Compatibility

• License: MIT License.
• Compatibility: Permissive MIT license allows for commercial use and integration with closed-source applications. Voicevox voices may have specific terms of use.

Limitations & Caveats

WhisperAI's accuracy is not guaranteed to be 100%, and it may struggle with multiple concurrent requests. Antivirus software might block ngrok/localtunnel connections used for cloud deployments. The current TTS implementation is primarily focused on Japanese (Voicevox) and German (Thorsten), requiring manual replacement for other languages.