This repository serves as a curated survey and resource hub for advancements in speech and audio large language models (LLMs). It categorizes and lists key research papers, models, and challenges across three core areas: speech representation learning, neural audio codecs, and speech LLMs themselves. The project is primarily aimed at researchers and engineers working in speech processing, natural language processing, and artificial intelligence, providing a comprehensive overview of the rapidly evolving landscape of spoken language technologies.

How It Works

The project functions as a living bibliography, meticulously tracking and organizing research papers, model releases, and relevant benchmarks in the speech/audio LLM domain. It categorizes contributions into distinct areas: learning discrete speech tokens for representation, developing neural codecs for efficient audio compression and reconstruction, and applying language modeling techniques to these tokens for speech understanding and generation tasks. This structured approach allows for a clear understanding of the interconnectedness and progression of these technologies.

Quick Start & Requirements

This repository is a survey and does not have a direct installation or execution command. It links to external research papers and code repositories for specific models. Users will need to refer to individual linked projects for their respective setup instructions and dependencies, which often include Python, deep learning frameworks (PyTorch/TensorFlow), and potentially specialized hardware like GPUs.

Highlighted Details

• Comprehensive catalog of over 150 research papers and models in speech/audio LLMs, neural codecs, and representation learning, updated frequently.
• Features a dedicated section for neural audio codec models, detailing their advancements in low-bitrate compression and reconstruction.
• Includes a curated list of speech representation models, focusing on techniques for learning discrete speech tokens.
• Highlights recent news and developments, including survey papers and participation in challenges like Codec-SUPERB at SLT 2024.

Maintenance & Community

The project is actively maintained by a team of researchers including Kai-Wei Chang, Haibin Wu, and Hung-yi Lee, with contributions from others. It references talks and tutorials from major conferences like ICASSP and Interspeech, indicating strong engagement with the academic community. Related repositories and citation information are provided for further exploration.

Licensing & Compatibility

The repository itself is a survey and does not impose a license. However, it links to numerous external projects, each with its own licensing terms. Users must consult the licenses of individual linked code repositories for usage, distribution, and commercialization rights.

Limitations & Caveats

As a survey, this repository does not provide executable code or pre-trained models directly. Users must navigate to individual linked projects to access and utilize specific models, which may have varying levels of maturity, documentation, and licensing restrictions. The rapid pace of research means the information may require continuous updates.