pykan  by KindXiaoming

Kolmogorov-Arnold Networks (KANs) as a drop-in replacement for MLPs

created 1 year ago
15,812 stars

Top 3.1% on sourcepulse

GitHubView on GitHub
1 Expert Loves This Project
Project Summary

Kolmogorov-Arnold Networks (KANs) offer an alternative to Multi-Layer Perceptrons (MLPs) by placing activation functions on edges rather than nodes, grounded in the Kolmogorov-Arnold representation theorem. This architectural shift aims to improve model accuracy and interpretability, particularly for scientific computing tasks. The project provides a Python library for implementing and experimenting with KANs.

How It Works

KANs leverage the Kolmogorov-Arnold representation theorem, which states that any continuous function of several variables can be represented as a finite sum of continuous functions of one variable. Unlike MLPs, where activation functions are applied at nodes, KANs apply learnable activation functions to the edges connecting neurons. This allows KANs to potentially represent complex functions more efficiently and with greater interpretability than traditional MLPs.

Quick Start & Requirements

  • Installation: pip install pykan or pip install -e . after cloning the repository.
  • Prerequisites: Python 3.9.7+, PyTorch 2.2.2, NumPy, SciPy, Matplotlib, Scikit-learn, Pandas, Seaborn, PyYAML.
  • Resources: Examples are runnable on a single CPU in under 10 minutes; paper examples take under a day. PDE training can take hours to days on a CPU. GPUs are recommended for larger-scale tasks.
  • Documentation: https://github.com/KindXiaoming/pykan
  • Tutorials: https://github.com/KindXiaoming/pykan/tree/main/tutorials

Highlighted Details

  • KANs are designed for tasks requiring high accuracy and interpretability.
  • Includes model.plot() for visualization and model.prune() for interpretability-focused refinement.
  • Offers advice on hyperparameter tuning, emphasizing starting simple and gradually increasing complexity.
  • The library is primarily geared towards scientific computing and research, not necessarily out-of-the-box ML applications.

Maintenance & Community

The project is maintained by KindXiaoming. The author notes that the code was developed with small-scale math and physics examples in mind and may lack efficiency or reusability optimizations. Further development for efficiency is not a primary focus, with external projects like efficientkan and fouierkan suggested.

Licensing & Compatibility

The repository does not explicitly state a license in the provided README. Users should verify licensing for commercial or closed-source use.

Limitations & Caveats

The author acknowledges that KANs may not be a simple plug-in for general machine learning tasks and require significant hyperparameter tuning. Efficiency and reusability are noted as areas for improvement, and the current implementation is not optimized for large-scale ML. The applicability to LLMs is uncertain due to differing definitions of accuracy and interpretability.

Health Check
Last commit

6 months ago

Responsiveness

1 day

Pull Requests (30d)
1
Issues (30d)
3
Star History
262 stars in the last 90 days

Explore Similar Projects

Starred by Andrej Karpathy Andrej Karpathy(Founder of Eureka Labs; Formerly at Tesla, OpenAI; Author of CS 231n) and Georgios Konstantopoulos Georgios Konstantopoulos(CTO, General Partner at Paradigm).

mlx-gpt2 by pranavjad

0.5%
393
Minimal GPT-2 implementation for educational purposes
created 1 year ago
updated 1 year ago
Feedback? Help us improve.