This repository provides pre-trained models and code for ToD-BERT, a natural language understanding model specifically designed for task-oriented dialogue systems. It addresses the linguistic differences between general text and dialogues, offering improved performance on downstream tasks like intent recognition, dialogue state tracking, and response selection, particularly in low-data scenarios. The target audience includes NLP researchers and developers working on conversational AI and task-oriented dialogue systems.

How It Works

ToD-BERT is pre-trained on a unified corpus of nine task-oriented dialogue datasets. It incorporates user and system tokens into the masked language modeling objective and introduces a contrastive objective function to simulate response selection. This dual-objective approach allows ToD-BERT to better capture dialogue-specific linguistic patterns and conversational dynamics compared to general-purpose language models.

Quick Start & Requirements

• Install/Run: Load pre-trained models via Huggingface Transformers:

  from transformers import AutoTokenizer, AutoModel
  tokenizer = AutoTokenizer.from_pretrained("TODBERT/TOD-BERT-JNT-V1")
  tod_bert = AutoModel.from_pretrained("TODBERT/TOD-BERT-JNT-V1")
  

• Prerequisites: PyTorch, Huggingface Transformers. GPU recommended for training/inference.
• Resources: Pre-trained models are available for download. Training requires significant computational resources and dialogue datasets.
• Links: Paper

Highlighted Details

• Outperforms BERT on four downstream task-oriented dialogue applications.
• Demonstrates stronger few-shot learning capabilities, mitigating data scarcity.
• Supports multiple pre-trained versions: MLM-only and MLM+RCL objectives.
• Includes scripts for pre-training and fine-tuning on dialogue tasks.

Maintenance & Community

• The project was released in 2020. Contact the first author (cswu@salesforce.com) or create an issue for inquiries.

Licensing & Compatibility

• The repository does not explicitly state a license. The associated paper is published by the Association for Computational Linguistics. Compatibility for commercial use or closed-source linking is not specified.

Limitations & Caveats

The project was last updated in October 2020, indicating potential staleness. The license is not explicitly mentioned, which may pose a barrier for commercial adoption.