This repository provides a from-scratch implementation of the "Attention Is All You Need" Transformer architecture. It targets engineers and researchers seeking a clear, understandable reference for the seminal paper, offering a flexible and runnable codebase for learning and experimentation, with benefits including CPU/GPU compatibility and detailed logging.

How It Works

The project implements the Transformer architecture solely based on attention mechanisms, eschewing recurrence and convolutions. This design enables parallelized sequence processing, a key advantage over RNNs, and facilitates the creation of context-aware word representations by considering word relationships within a sequence.

Quick Start & Requirements

• Installation: Use Miniconda/Anaconda: conda env create followed by conda activate attention-is-all-you-need-paper.
• Prerequisites: Python. CUDA version 10.2 may be required depending on the GPU. The implementation is runnable on CPU.
• Documentation: A tutorial notebook (notebooks/tutorial.ipynb) provides a detailed walkthrough of the architecture.

Highlighted Details

• Runnable on both CPU and GPU.
• Integrated Weights and Biases (W&B) for real-time logging and visualization of training metrics.
• Highly customizable configuration for training loops, model dimensions, and tokenizers.
• Supports BPE and WordLevel Tokenizers, dynamic batching, and batch dataset processing.
• Includes Bleu-score calculation during training and documented architectural dimensions.

Maintenance & Community

The repository is maintained by Brando Koch. No specific community channels (e.g., Discord, Slack) or details on sponsorships/partnerships are provided in the README.

Licensing & Compatibility

The project is released under the MIT License, generally permitting commercial use and modification.

Limitations & Caveats

This implementation focuses on the original Transformer architecture for sequence-to-sequence tasks like machine translation and does not cover encoder-only (e.g., BERT) or decoder-only (e.g., GPT) variants. It is primarily intended for learning purposes.