This library provides Keras layers for building Universal Transformer models, targeting researchers and practitioners in NLP. It offers a flexible, modular approach to constructing Transformer architectures, enabling experimentation with models like BERT and GPT.

How It Works

The library implements core Transformer components as standalone Keras layers, including positional encoding, attention masking, and memory-compressed attention. This modular design allows users to assemble custom Transformer architectures by composing these layers, facilitating experimentation with variations like Adaptive Computation Time (ACT) and enabling direct replacement or rearrangement of components.

Quick Start & Requirements

• Install via pip install . after cloning the repository.
• Requires Python >= 3.6.
• Example usage requires pip install -r example/requirements.txt and a Keras backend like TensorFlow (pip install tensorflow).
• Examples are available for BERT and GPT language modeling on WikiText-2.

Highlighted Details

• Supports Universal Transformers, BERT, and GPT architectures.
• Includes memory-compressed attention and Adaptive Computation Time (ACT).
• Modular Keras layers allow for custom model construction.
• Demonstrates language modeling on WikiText-2 with perplexity metrics.

Maintenance & Community

No specific information on maintainers, community channels, or roadmap is provided in the README.

Licensing & Compatibility

The README does not explicitly state a license.

Limitations & Caveats

The provided examples are demonstrations and not rigorous evaluations. Training BERT models requires significant time and computational resources.