SoulX-Duplug provides a plug-and-play streaming semantic Voice Activity Detection (VAD) module designed for real-time, full-duplex spoken dialogue systems. It addresses the need for low-latency, semantically aware interaction by enabling real-time state prediction based on audio input. This project is targeted at researchers and developers building advanced conversational AI, offering a practical solution to enhance the responsiveness and naturalness of speech-based interactions.

How It Works

The core innovation lies in "text-guided streaming state prediction." SoulX-Duplug processes audio in chunks, using Automatic Speech Recognition (ASR) results and a short audio buffer to predict dialogue states: "idle" (silence/noise), "nonidle" (semantic content detected), "speak" (utterance complete, system can respond), or "blank" (incomplete chunk). This approach allows for semantic understanding and low-latency turn-taking crucial for full-duplex conversations, differentiating it from traditional VAD methods by incorporating semantic context.

Quick Start & Requirements

• Primary install: Clone the repository, install system dependencies (ffmpeg, sox, libsox-dev), create and activate a Conda environment (conda create -n soulx-duplug python=3.10, conda activate soulx-duplug), then run pip install -r requirements.txt.
• Non-default prerequisites: Linux OS, Conda, ffmpeg, sox, git-lfs (for model download via git).
• Model Download: Checkpoints are available via Hugging Face CLI (huggingface-cli download), Python (snapshot_download), or git clone (requires git-lfs). Models are located at Soul-AILab/SoulX-Duplug-0.6B.
• Configuration: Adjust ASR models (Paraformer for Chinese, SenseVoice for English/auto) and parameters like max_wait_num and far_field_threshold in config/config.yaml.
• Basic Usage: Start the streaming inference server using bash run.sh.
• Links:
  • Demo Video: https://github.com/user-attachments/assets/cf5b040e-bc87-4fa9-ae6f-669db80a49eb
  • Online Demo: https://soulx-duplug.sjtuxlance.com/
  • Paper: https://arxiv.org/abs/2603.14877
  • Hugging Face Models: Soul-AILab/SoulX-Duplug-0.6B

Highlighted Details

• Enables plug-and-play streaming semantic VAD for real-time full-duplex speech interaction.
• Features text-guided streaming state prediction for low-latency, semantic-aware dialogue management.
• Includes a demo dialogue system and the SoulX-Duplug-Eval dataset for benchmarking full-duplex systems.
• Supports Chinese, English, and bilingual speech input with configurable ASR backends.

Maintenance & Community

The project's paper was published in March 2026, with model checkpoints and evaluation data released concurrently on Hugging Face. The README does not specify community channels (e.g., Discord, Slack) or a public roadmap. Acknowledgments are made to several open-source projects.

Licensing & Compatibility

This project is licensed under the Apache 2.0 License. This license is permissive and generally compatible with commercial use and linking within closed-source applications.

Limitations & Caveats

Installation instructions are primarily detailed for Linux environments. The project focuses on streaming inference and state prediction, with the core dialogue system implementation available on a separate branch. No explicit mention of alpha/beta status, known bugs, or unsupported platforms beyond the Linux installation focus is provided.