Kernl is an open-source Python library designed to accelerate PyTorch transformer model inference on GPUs by several times. It targets researchers and engineers working with large language models who need to improve inference speed and reduce latency, offering a more hackable alternative to traditional inference engines.

How It Works

Kernl leverages OpenAI Triton, a Python-based language for writing GPU kernels, to rewrite critical operations like attention, linear layers, and layernorm. This approach allows for operator fusion, reducing memory bandwidth bottlenecks by avoiding intermediate results storage. It also utilizes CUDA graphs for zero-overhead inference replay and TorchDynamo to handle dynamic model behaviors by tracing and recompiling optimized computation graphs.

Quick Start & Requirements

• Install via pip: pip install 'git+https://github.com/ELS-RD/kernl'
• Requires PyTorch, Python >= 3.9, an Ampere GPU, and CUDA.
• Docker image available for easier setup.
• See Examples for end-to-end use cases.

Highlighted Details

• Achieves significant speedups on transformer models like Llama v2, T5, and Whisper.
• Kernels are written in OpenAI Triton, with individual kernels under 200 lines of code for ease of modification.
• Supports optimization through kernel fusion and replacement of PyTorch operations with custom Triton kernels.
• Includes extensive benchmarking tools and conventions for performance analysis.

Maintenance & Community

• Developed by ELS-RD.
• Contribution guide and Code of Conduct are available.

Licensing & Compatibility

• License not explicitly stated in the README.

Limitations & Caveats

• Requires specific hardware (Ampere GPU) and CUDA installation.
• Benchmarks can take a considerable amount of time to run.
• The project is built on newer technologies like Triton and TorchDynamo, which may still be evolving.