NMT chatbot using seq2seq for sentence translation
Top 74.9% on sourcepulse
This project implements a Neural Machine Translation (NMT) chatbot, offering sentence translation capabilities alongside its primary chatbot function. It targets developers and researchers interested in building conversational AI using seq2seq models, providing a custom BPE/WPM-like tokenizer and an embedded detokenizer for improved vocabulary management and translation quality.
How It Works
The chatbot leverages a seq2seq architecture built upon a fork of the NMT framework. It features a custom implementation of BPE/WPM-like subword tokenization, which allows for a smaller vocabulary size by breaking down rare words into common subword units. This approach aims to reduce <unk>
tokens and improve translation accuracy. An embedded detokenizer is used, simplifying the detokenization process by avoiding rule-based systems.
Quick Start & Requirements
pip install -r requirements.txt
after cloning the repository.new_data
folder.Highlighted Details
Maintenance & Community
The project is a fork of NMT, with modifications to support TensorFlow 1.4. Recent updates include a new response scoring engine, performance improvements for data preparation, and relative model paths for easier deployment.
Licensing & Compatibility
The README does not explicitly state a license. Compatibility for commercial use or closed-source linking is not specified.
Limitations & Caveats
The project requires specific, older versions of TensorFlow, CUDA, and cuDNN, which may pose compatibility challenges with modern hardware and software stacks. The fork nature means maintaining synchronization with upstream NMT changes could be a burden.
5 years ago
1 day