A simplified library for decentralized, privacy-preserving machine learning, Swarm Learning enables training ML models at the edge where data resides. It targets users requiring enhanced data security and privacy by sharing only learned insights, not raw data, across distributed nodes. The framework leverages blockchain for secure aggregation of learnings, maximizing the utility of distributed computing power.

How It Works

Swarm Learning employs a decentralized architecture composed of modular, containerized nodes: Swarm Learning (SL) nodes for core ML training and weight management, Swarm Network (SN) nodes forming an Ethereum-based blockchain for coordination and metadata tracking, Swarm Operator (SWOP) nodes for task execution, and Swarm Learning Command Interface (SWCI) nodes for management. Communication is secured via X.509 certificates, with optional integration for SPIFFE/SPIRE. User ML programs (Keras, PyTorch, HuggingFace) are adapted using a SwarmCallback API, with ML and SL nodes running collaboratively on the same host.

Quick Start & Requirements

• Installation: Primarily via the SLM-UI installer.
• Prerequisites: X.509 certificates are mandatory for framework initialization. Participating nodes require network access to each other's ports. A license managed by AutoPass License Server (APLS) is required.
• Examples: Includes MNIST and mini LLM fine-tuning (HuggingFace Trainer LoRA).
• Documentation: Detailed documentation is available, covering concepts, configuration, and usage. Links to SPIFFE/SPIRE resources are provided.

Highlighted Details

• Privacy-Preserving: Achieves privacy by sharing only model learnings, never raw data.
• Decentralized Training: Utilizes computing power at distributed data sources (edge computing).
• Blockchain Integration: Employs an Ethereum-based blockchain for secure coordination and progress tracking.
• Framework Compatibility: Supports Keras, PyTorch, and HuggingFace Trainer classes, requiring consistency across nodes.

Maintenance & Community

The project is maintained by HewlettPackard. Bug reporting and contributions are managed via the GitHub issue tracker and contributing guide.

Licensing & Compatibility

The distribution is licensed for non-commercial and experimental use only. Specific terms for datasets are detailed in ATTRIBUTIONS and DATA LICENSE. Commercial use or closed-source linking may be restricted.

Limitations & Caveats

Certain features, such as "Accessing Hewlett Packard Enterprise Support" and "Concurrent swarm training," are exclusively for enterprise customers. Bundled examples may not reflect the latest code; direct GitHub copies are recommended for troubleshooting. All participating ML nodes must adhere to a single ML platform (Keras, PyTorch, or HuggingFace).