An end-to-end framework for multi-speaker transcription, SoulX-Transcriber directly models speaker attribution, timestamped segmentation, and transcription in a single unified model. It addresses challenges in multi-speaker dialogue scenarios, including overlapping speech and fast turns, by producing coherent, speaker-consistent transcripts. This unified approach benefits researchers and developers requiring robust diarization and recognition capabilities.

How It Works

The core of SoulX-Transcriber is a unified end-to-end large audio language model that jointly learns speaker attribution, precise timestamped segmentation, and transcription, eliminating the need for cascaded pipelines. It employs speaker-aware multi-stage training, combining continued pre-training with supervised fine-tuning, to enhance speaker representation and robustness against issues like same-gender confusion, speech overlap, and boundary errors. Additionally, a novel speaker characteristics-driven audio matching pipeline is proposed for dialogue simulation, automatically selecting optimal reference audio for each utterance to generate natural, context-aligned simulated dialogues.

Quick Start & Requirements

• Installation: Clone the repository, create a Conda environment with Python 3.12, activate it, and install ms-swift and its dependencies. Inference requires setting up vLLM-omni with Python 3.12.
• Prerequisites: Python 3.12, Conda, ms-swift, vLLM-omni.
• Model Download: Pre-trained model weights are available on Hugging Face and ModelScope for ZH/EN languages.
• Documentation: Links to demo pages and vLLM official documentation for compilation are provided.

Highlighted Details

• Achieves state-of-the-art performance on AISHELL-4 and AliMeeting benchmarks via its unified diarization and recognition framework.
• Employs speaker-aware multi-stage training (pre-training + fine-tuning) to bolster speaker representation and robustness against common diarization errors.
• Features a multi-speaker dialogue simulation pipeline with a speaker-aware prompt audio matching mechanism for improved out-of-domain generalization.

Maintenance & Community

Contact information for issues and inquiries is provided via GitHub Issues and specific email addresses. No explicit links to community forums (e.g., Discord, Slack) or detailed roadmaps are present in the README.

Licensing & Compatibility

The project is released under the Apache 2.0 License. This license permits free use by researchers and developers for code and model weights, with no explicit restrictions mentioned for commercial use or closed-source linking beyond standard Apache 2.0 terms.

Limitations & Caveats

The README does not explicitly detail known limitations, alpha status, or specific unsupported platforms. The inference setup relies on vLLM-omni, which may have its own system requirements and potential build complexities.