Confucius4-TTS is an advanced LLM-based text-to-speech (TTS) system designed for multilingual and cross-lingual speech synthesis. It enables high-quality speech generation while preserving speaker identity across languages, targeting researchers and developers needing flexible, zero-shot TTS capabilities. The key benefit is its ability to clone voices and transfer them across 14 languages without additional training.

How It Works

The system employs a "speech encoder + large language model (LLM)" architecture. It comprises two main modules: Text2Semantic (T2S), which generates semantic token sequences from text and speaker conditioning, and Semantic2Acoustic (S2A), a flow-matching model that converts these semantic tokens into mel spectrograms. This approach allows for unconstrained voice cloning and seamless cross-lingual voice transfer, preserving speaker identity and emotion.

Quick Start & Requirements

• Installation: Clone the repository, create a conda environment with Python 3.10, and install dependencies via pip install -r requirements.txt.
• Prerequisites: Python 3.10, CUDA 12.6.
• Demo: An online demo is available at https://confucius4-tts.youdao.com/gradio.

Highlighted Details

• Supports 14 languages including Chinese, English, Japanese, Korean, German, French, Spanish, Indonesian, Italian, Thai, Portuguese, Russian, Malay, and Vietnamese.
• Features unconstrained voice cloning, requiring no reference transcript.
• Enables cross-lingual voice transfer, synthesizing speech in one language using a voice from another.
• Supports seamless emotion transfer, cloning the emotional tone of the reference voice.
• Demonstrates robust generalization in real-world multilingual scenarios.

Maintenance & Community

The README does not provide specific details on maintenance, community channels (like Discord/Slack), or a roadmap.

Licensing & Compatibility

The README does not specify the software license or any compatibility notes for commercial use.

Limitations & Caveats

The README does not explicitly state limitations or caveats. However, performance tables indicate that while Confucius4-TTS achieves competitive results, other models may outperform it on specific metrics (e.g., WER, SIM) in certain cross-lingual directions or require reference text.