This repository is a curated list of papers, implementations, and resources related to Knowledge Distillation (KD) in machine learning. It serves as a comprehensive reference for researchers and practitioners interested in model compression, knowledge transfer, and improving the efficiency of neural networks by training smaller "student" models to mimic larger "teacher" models.

How It Works

The repository organizes a vast collection of academic papers, categorizing them by year and topic. It also links to various open-source implementations of KD techniques across different deep learning frameworks like PyTorch, TensorFlow, MXNet, and Caffe. This allows users to explore the theoretical foundations and practical applications of KD methods, from foundational concepts like "dark knowledge" to advanced techniques such as contrastive distillation and data-free distillation.

Highlighted Details

• Extensive bibliography covering KD from 1990 to the present, including seminal works by Hinton, Caruana, and Vapnik.
• Links to numerous code repositories implementing diverse KD strategies, including attention transfer, feature distillation, and adversarial distillation.
• Coverage of KD applications in various domains, such as computer vision, natural language processing, and speech synthesis.
• Includes resources for specific frameworks and advanced topics like distilling diffusion models and LLMs.

Maintenance & Community

This is a community-driven "awesome" list, meaning it is maintained by contributions from users. There are no specific contributors or community channels explicitly mentioned in the README.

Licensing & Compatibility

The repository itself is a list of links and does not have a specific license. The linked papers and code repositories are subject to their respective licenses.

Limitations & Caveats

As a curated list, the repository's content is dependent on community contributions and may not be exhaustive or perfectly up-to-date. The quality and maintenance status of linked code implementations can vary.