This repository provides a comprehensive collection of optimizers, learning rate schedulers, and loss functions for PyTorch, aiming to simplify and enhance deep learning model training. It is designed for researchers and practitioners seeking to experiment with a wide array of optimization techniques beyond standard offerings, potentially improving convergence speed and model generalization.

How It Works

The library offers a unified interface to over 100 optimizers, 16 LR schedulers, and 13 loss functions. It integrates popular and novel algorithms, including variants of Adam, SGD, and Sharpness-Aware Minimization (SAM), along with specialized optimizers like Lion and Prodigy. The core design emphasizes ease of use, allowing users to load optimizers by name or through a create_optimizer function, and supports integration with libraries like bitsandbytes for 8-bit optimization.

Quick Start & Requirements

• Install via pip: pip3 install pytorch-optimizer
• Supports Python 3.8+ (Python 3.7 can be used with --ignore-requires-python).
• Optional dependencies for bitsandbytes, q-galore-torch, and torchao require separate installation.
• Usage examples and detailed documentation are available.

Highlighted Details

• Supports 102+ optimizers, 16 LR schedulers, and 13 loss functions.
• Includes advanced techniques like Sharpness-Aware Minimization (SAM), Gradient Centralization (GC), and Adaptive Gradient Clipping (AGC).
• Offers integration with bitsandbytes for 8-bit optimizers and memory-efficient training.
• Provides a torch.hub loading option for easy experimentation.

Maintenance & Community

• Actively maintained by Hyeongchan Kim (@kozistr).
• The repository includes a citation for the software itself.

Licensing & Compatibility

• Most optimizers are MIT or Apache 2.0 licensed.
• Some optimizers (e.g., Fromage, Nero) have CC BY-NC-SA 4.0 licenses, which are non-commercial. Users must verify licenses before commercial use.

Limitations & Caveats

The library includes optimizers with non-commercial licenses, requiring careful attention from users intending to use them in commercial projects.