This repository provides a Keras implementation of Google's BERT and OpenAI's Transformer LM, enabling transfer learning with pre-trained weights. It offers a flexible API for fine-tuning models on various NLP tasks, including sentence-level (e.g., classification) and token-level (e.g., PoS tagging) tasks, abstracting data generation and task management.

How It Works

The library utilizes a modular design, separating text encoding, language model generation, and task definitions. Users can define custom encoders (e.g., LSTMs) or leverage included implementations. A TaskMetaDatas structure handles task-specific labels and masking, facilitating both pre-training and fine-tuning. The train_model function orchestrates the process, supporting flexible training schedules via TaskWeightScheduler and custom attention masks for controlling model behavior.

Quick Start & Requirements

• Install: pip install keras (backend: TensorFlow recommended)
• Prerequisites: Python, Keras, TensorFlow (for BERT loading, training, and fine-tuning). Theano is supported for OpenAI model loading.
• Resources: Requires pre-trained model weights.
• Docs: tutorial.ipynb, Colab notebooks linked in README.

Highlighted Details

• Supports both BERT and OpenAI Transformer LM architectures.
• Abstracted data generation and task management for NLP tasks.
• TPU support for inference and training.
• Allows custom encoders (e.g., BiLSTM) if they adhere to the transformer encoder's input/output contract.

Maintenance & Community

• Status: Archived; no further updates expected.
• Contributors: Mentions @HighCWu for TPU support.
• Ownership: Neiron.

Licensing & Compatibility

• License: Not explicitly stated in the README.
• Compatibility: TensorFlow backend recommended for BERT features. Theano backend has limitations for BERT model loading and training.

Limitations & Caveats

The project is archived and will not receive further updates. BERT model loading and training are not supported on the Theano backend. The tutorial.ipynb notebook is described as poorly designed.