Native Diffusion enables local Stable Diffusion inference on iOS and macOS using Swift and Apple's MPSGraph framework, eliminating the need for Python environments. It offers a simple, flexible API for developers targeting Apple platforms, allowing for image generation and image-to-image transformations with minimal setup.

How It Works

This project leverages MPSGraph, Apple's Metal Performance Shaders Graph API, to execute Stable Diffusion models natively on Apple silicon. It converts standard .ckpt model files into a memory-optimized format, utilizing FP16 tensors and operator fusion for efficient execution. This approach aims for performance comparable to or exceeding cloud-based solutions while ensuring a seamless, zero-dependency user experience on end-user devices.

Quick Start & Requirements

• Install: Add https://github.com/mortenjust/native-diffusion to Xcode's Swift Package Manager.
• Prerequisites: macOS or iOS device. Python 3.10 with torch, typing_extensions, numpy, Pillow, requests, and pytorch_lightning is required for model conversion.
• Model Conversion: A Python script (maple-convert.py) is provided to convert .ckpt files to the native format.
• Resources: Model files can be downloaded from a URL or included in the app bundle.
• Docs: https://github.com/mortenjust/native-diffusion

Highlighted Details

• 100% native Swift implementation, no Python dependencies for inference.
• Supports image-to-image transformations and Dreambooth models.
• Offers Combine publishers and async/await for integration.
• Claims performance faster than Python/PyTorch/Tensorflow on M1 Max Macs.

Maintenance & Community

• Active development with a roadmap including Stable Diffusion 2.0, in-painting, and on-device training.
• Community engagement via Discord.

Licensing & Compatibility

• Code is MIT Licensed.
• Generated images are under the CreativeML Open RAIL-M license, permitting commercial use.

Limitations & Caveats

• Stable Diffusion 2.0 and certain advanced features like in-painting are not yet supported.
• The project is described as largely proof-of-concept, with ongoing efforts to improve code quality.