EET (Easy and Efficient Transformer) is a PyTorch inference plugin designed to optimize the performance and affordability of large Transformer-based NLP and multi-modal models. It targets researchers and developers working with models like GPT-3, BERT, CLIP, Baichuan, and LLaMA, offering significant speedups and reduced memory footprints for single-GPU inference.

How It Works

EET achieves its performance gains through a combination of CUDA kernel optimizations and quantization/sparsity algorithms. It provides low-level "Operators APIs" that can be composed to build custom model architectures, as well as higher-level "Model APIs" that seamlessly integrate with Hugging Face Transformers and Fairseq models. This layered approach allows for both deep customization and easy adoption.

Quick Start & Requirements

• Installation: Recommended via Docker (docker build -t eet_docker:0.1 . then nvidia-docker run ...). Alternatively, clone the repo and pip install . from source.
• Prerequisites: CUDA >= 11.4, Python >= 3.7, GCC >= 7.4.0, PyTorch >= 1.12.0, NumPy >= 1.19.1, Fairseq == 0.10.0, Transformers >= 4.31.0.
• Resources: Supports mega-size models on a single GPU.
• Docs: https://github.com/NetEase-FuXi/EET/tree/main/example/python

Highlighted Details

• Supports int8 quantization for further optimization.
• Offers out-of-the-box integration with Transformers and Fairseq.
• Provides ready-made pipelines for common NLP tasks (text-classification, fill-mask, text-generation, etc.).
• Claims 2-8x speedup for models like GPT-3 and T5.

Maintenance & Community

• Contact via GitHub issues or email (zhaosida@corp.netease.com, zhuangzhong@corp.netease.com, hzzhaozeng@corp.corp.netease.com).
• A video presentation is available: https://event.baai.ac.cn/activities/325.

Licensing & Compatibility

• The repository does not explicitly state a license in the README. Compatibility for commercial use or closed-source linking is not specified.

Limitations & Caveats

• The project is described as "beta" in some sections.
• Only pre-padding is supported for GPT-3.
• The license is not clearly defined, which may impact commercial adoption.