SenseVoice is a multilingual speech foundation model offering Automatic Speech Recognition (ASR), Spoken Language Identification (LID), Speech Emotion Recognition (SER), and Audio Event Detection (AED). It targets developers and researchers needing high-accuracy, low-latency speech processing across multiple languages, providing a significant performance uplift over models like Whisper.

How It Works

SenseVoice employs a non-autoregressive end-to-end framework for efficient inference. It is trained on over 400,000 hours of multilingual data, enabling robust performance across its diverse speech understanding capabilities. The model architecture is designed for low latency, making it suitable for real-time applications.

Quick Start & Requirements

• Install via pip: pip install -r requirements.txt
• Requires Python and potentially CUDA for GPU acceleration.
• Official Docs: Install, Usage
• Model Zoo: ModelScope, Hugging Face
• Online Demos: ModelScope, Hugging Face

Highlighted Details

• Multilingual ASR: Supports over 50 languages, outperforming Whisper on benchmark datasets.
• Advanced SER and AED: Achieves state-of-the-art results in emotion and audio event detection.
• Efficient Inference: SenseVoice-Small processes 10 seconds of audio in 70ms, 15x faster than Whisper-Large.
• Exportable: Supports ONNX and Libtorch formats for broader deployment.
• Finetuning: Provides scripts and strategies for custom model adaptation.

Maintenance & Community

• Active development with recent updates in July and November 2024.
• Community support via GitHub Issues and DingTalk group.
• Related projects include FunASR, CosyVoice, and SenseVoice.cpp.

Licensing & Compatibility

• The specific license is not explicitly stated in the README, but it is associated with Alibaba's iic/FunAudioLLM. Compatibility for commercial use should be verified.

Limitations & Caveats

• Audio Event Detection performance has some gaps compared to specialized AED models due to training data limitations.
• Pseudo-streaming via streaming-sensevoice sacrifices some accuracy for lower latency.