This library provides a PyTorch implementation of the learning rate range test, a technique for identifying optimal learning rates for neural network training. It's designed for researchers and practitioners seeking to improve model convergence and stability by systematically exploring learning rate boundaries.

How It Works

The library offers two primary methods: an exponential increase in learning rate (similar to fastai) or a linear increase (Leslie Smith's original approach). The exponential method computes training loss per learning rate, while the linear method uses evaluation loss. The exponential method is faster, while the linear method can yield more precise curves by leveraging evaluation loss sensitivity to divergence.

Quick Start & Requirements

• Install via pip: pip install torch-lr-finder
• Requires Python 3.5+ and PyTorch.
• GPU with CUDA is recommended for performance.
• Official examples for CIFAR10 and MNIST are available in the examples folder.

Highlighted Details

• Supports both fastai-style exponential and Leslie Smith-style linear learning rate scheduling.
• Integrates with NVIDIA Apex and PyTorch's native AMP for mixed-precision training.
• Includes gradient accumulation support for training with larger effective batch sizes.
• Allows access to learning rate and loss history via lr_finder.history.

Maintenance & Community

• The repository is actively maintained by davidtvs.
• Contributions and pull requests are welcomed, with guidelines provided in CONTRIBUTING.md.

Licensing & Compatibility

• The license is not explicitly stated in the README. Compatibility for commercial use or closed-source linking is therefore unclear.

Limitations & Caveats

The README does not specify the license, which may impact commercial adoption. The effectiveness of mixed-precision training is dependent on specific NVIDIA GPU hardware.