YouDub-webui is a Gradio-based web UI for the YouDub project, designed to translate and dub YouTube videos into Chinese using AI. It targets users who want to localize foreign video content for educational, entertainment, or professional purposes, offering an accessible interface for non-technical users.

How It Works

The tool orchestrates a pipeline of AI models for video localization. It begins with video downloading, followed by AI-powered speech recognition (using WhisperX for accurate transcription and speaker diarization), large language model-based translation (leveraging OpenAI GPT and other LLMs), and AI voice cloning (utilizing Coqui AI TTS and Volcano Engine TTS for natural-sounding dubbing). Finally, it handles video processing for audio-video synchronization, subtitle embedding, and optional auto-upload to Bilibili.

Quick Start & Requirements

• Install: Clone the repository and run setup_windows for automatic dependency installation (including CUDA 12.1 PyTorch) or pip install -r requirements.txt followed by pip install TTS for manual setup.
• Prerequisites: Python, Git, OpenAI API Key, Hugging Face Token, Volcano Engine TTS credentials, Bilibili credentials. CUDA 12.1 is recommended for PyTorch.
• Setup: Requires environment variable configuration (.env file).
• Run: Execute run_windows.bat or python app.py.
• Docs: bilibili video page

Highlighted Details

• Integrates WhisperX for advanced speech recognition with speaker diarization.
• Supports multiple LLMs for translation via OpenAI API and api-for-open-llm.
• Employs Coqui AI TTS and Volcano Engine TTS for AI voice cloning.
• Features an "All-in-One" mode for automated end-to-end video localization.
• Includes an option for automatic video upload to Bilibili.

Maintenance & Community

• Community support is available via Discord server and WeChat groups.
• Contributions are welcomed via GitHub Issues and Pull Requests.

Licensing & Compatibility

• Licensed under Apache License 2.0.
• Users must comply with copyright, data protection, and privacy laws. Unauthorized use without content creator permission is prohibited.

Limitations & Caveats

The project primarily targets Windows users with the provided scripts. Manual installation of specific PyTorch CUDA versions may be required if the automatic setup does not match the user's environment. Credentials for various services (OpenAI, Hugging Face, Volcano Engine, Bilibili) are necessary for full functionality.