ConvLab-2 is an open-source toolkit for building, evaluating, and diagnosing task-oriented dialogue systems. It supports state-of-the-art models and multiple datasets, offering researchers tools for end-to-end evaluation and system weakness analysis.

How It Works

ConvLab-2 provides a modular framework allowing users to integrate various Natural Language Understanding (NLU), Dialogue State Tracking (DST), Policy, and Natural Language Generation (NLG) models. It supports pipeline and end-to-end approaches, facilitating experimentation with different model combinations and offering a robust evaluation suite with user simulators.

Quick Start & Requirements

• Install via pip: pip install -e . after cloning the repository.
• Requires Python >= 3.6.
• Supports multiple datasets including MultiWOZ, CrossWOZ, Camrest, and Dealornot.
• Tutorials and documentation are available online.

Highlighted Details

• Integrates transformer-based models for NLU, DST, and policy modules.
• Includes transferable user simulators adaptable to custom datasets.
• Offers an advanced RL toolkit for policy training and evaluation.
• Supports end-to-end evaluation on MultiWOZ with detailed performance metrics.

Maintenance & Community

Maintained by Tsinghua University Conversational AI group (THU-coai) and Microsoft Research (MSR). ConvLab-3 was released in November 2022, indicating ongoing development.

Licensing & Compatibility

Licensed under Apache License 2.0. Datasets have specific Creative Commons licenses (e.g., CC BY 4.0, CC BY-NC 4.0), with the latter restricting commercial use.

Limitations & Caveats

Trained models hosted on Azure were temporarily inaccessible due to security risks, with a migration to Hugging Face. The README notes that end-to-end performance results are for commits before a specific hash and may be updated.