FlashCosyVoice is a lightweight, pure Python implementation of CosyVoice designed for high-speed offline batch inference. It targets researchers and developers needing efficient voice generation without the complexities of standard vLLM, offering significant speedups and easier customization.

How It Works

This project reimplements CosyVoice's inference pipeline from scratch, integrating optimizations like prefix caching, Torch compilation, and CUDA graphs directly. This approach bypasses the need for external vLLM installations, achieving up to a 9x speedup over native PyTorch inference while maintaining comparable or improved Word Error Rates (WERs). The clean, minimal Python codebase facilitates easier modification and integration of custom sampling methods.

Quick Start & Requirements

• Installation:
  • For direct use: pip install git+https://github.com/xingchensong/FlashCosyVoice
  • For modification: git clone https://github.com/xingchensong/FlashCosyVoice && cd FlashCosyVoice && pip install -e .
• Model Download: Requires git-lfs (sudo apt-get install git-lfs) and then git clone https://www.modelscope.cn/iic/CosyVoice2-0.5B.git.
• Dependencies: Python, git-lfs. Specific hardware (e.g., GPU) is implied by the example usage and performance benchmarks.
• Example Usage: Distributed batch inference using torchrun, with parameters for model path, data list, batch sizes, workers, and precision (fp16_flow).
• Documentation: No explicit links to separate docs or demos are provided in the README.

Highlighted Details

• Achieves up to 9x speedup over native PyTorch inference (RTF of 0.055 with bf16 LLM + fp16 flow) while maintaining similar WERs.
• Offers superior stability and WER performance compared to cosyvoice/vllm_example.py due to its implementation of ras_sample.
• Optimized for offline batch inference, supporting configurations from 1 GPU to multi-GPU setups.
• Leverages techniques like prefix caching, Torch compilation, and CUDA graphs for performance gains.

Maintenance & Community

The README does not mention specific contributors, sponsorships, or community channels (like Discord/Slack). It acknowledges inspiration from nano-vllm and other CosyVoice-related projects.

Licensing & Compatibility

The license type is not specified in the README. Compatibility for commercial use or closed-source linking cannot be determined without a license.

Limitations & Caveats

• Currently lacks support for online generation required for RL training.
• Future support for Ray for large-scale generation and CosyVoice3 is listed as a TODO.
• The absence of a specified license prevents definitive statements on commercial use.