OpenNMT-py is an open-source framework for neural machine translation and large language models, built with PyTorch. It targets researchers and developers seeking to experiment with and deploy advanced NLP models, offering production-ready capabilities and extensive customization for tasks like translation, summarization, and language modeling.

How It Works

The framework leverages PyTorch for its neural network implementations, supporting various attention mechanisms like Multihead and Multiquery attention. It integrates advanced features such as 8-bit and 4-bit quantization (including AWQ), LoRA adapters, and tensor parallelism for efficient training and inference of large models on limited hardware.

Quick Start & Requirements

• Install: pip install OpenNMT-py or pip install -e . from source.
• Prerequisites: Python >= 3.8, PyTorch >= 2.0 <2.2. For optimal performance and advanced features, installation of Apex and Flash Attention 2 is recommended.
• Docker: Pre-built images are available on GitHub Container Registry: ghcr.io/opennmt/opennmt-py.
• Docs: Full HTML Documentation

Highlighted Details

• Supports LLM fine-tuning (e.g., Llama, Mistral) on single GPUs with 4-bit quantization.
• Offers efficient inference via CTranslate2.
• Includes tutorials for replicating models like Vicuna and fine-tuning NLLB-200.
• Features Multiquery attention for improved training and inference speed.

Maintenance & Community

• Announcement: The project is no longer actively supported; a new project, Eole, has been initiated.
• Community: Forum, Gitter.
• Maintainers: François Hernandez, Vincent Nguyen (Seedfall).

Licensing & Compatibility

• License: MIT.
• Compatibility: Permissive license suitable for commercial use and integration into closed-source projects.

Limitations & Caveats

The project is no longer actively supported, with a recommendation to migrate to the new Eole project. While production-ready features exist, ongoing development and support have ceased for OpenNMT-py itself.