This repository provides tools and pre-trained models for Russian GPT-2 language generation, targeting researchers and developers interested in fine-tuning or deploying large language models for Russian text. It offers a comprehensive guide for training, evaluation, and deployment, including performance benchmarks and detailed instructions for dataset preparation and model configuration.

How It Works

The project leverages the GPT-2 architecture and implements progressive layer unfreezing for efficient transfer learning. It utilizes a custom YTTM tokenizer, noted for its speed and smaller file sizes compared to SentencePiece. Training is optimized with mixed-precision (fp16) and supports both GPU and Google TPU acceleration.

Quick Start & Requirements

• Install: conda env create -f environment.yml
• Prerequisites: Python, CUDA (version 10.2 mentioned for Apex), AWS CLI, apt.txt dependencies.
• Resources: Training on large datasets can be resource-intensive, with TPU v3-8 being significantly faster than a Titan RTX.
• Links:
  • Fine-tuning Colab: https://colab.research.google.com/drive/1jwFks82BLyy8x3oxyKpiNdlL1PfKSQwW
  • Corpus Generation Colab: https://colab.research.google.com/drive/1Hsp2508TXMR0ihYOLjKYOzWm9byqg9ue
  • Model Downloads: https://s3.amazonaws.com/models.dobro.ai/gpt2/ru/index.html

Highlighted Details

• Perplexity benchmarks provided for various model sizes (124M, 355M) and training configurations on different Russian datasets.
• Supports gradual unfreezing strategy (0, 1, 2, 7, -1) for progressive training.
• Includes scripts for model evaluation, text processing, and token conversion.
• Offers a REST API deployment example using uvicorn.

Maintenance & Community

• The project appears to be maintained by mgrankin.
• Links to Telegram bots (@PorfBot, @NeuroPoetBot) for direct model interaction are provided.

Licensing & Compatibility

• The repository does not explicitly state a license in the README.
• Compatibility for commercial use or closed-source linking is not specified.

Limitations & Caveats

• The README mentions potential issues with Apex and DataParallel (apex/issues/227), which might affect mixed-precision training on certain configurations.
• Instructions for SentencePiece installation are provided but noted as skippable if using YTTM.
• The project relies on AWS S3 for model distribution.