SegMoE provides a framework for dynamically combining Stable Diffusion models into a Mixture of Experts (MoE) without retraining. This allows users to create larger models with enhanced knowledge, better prompt adherence, and improved image quality, targeting users who want to leverage multiple fine-tuned models efficiently.

How It Works

SegMoE dynamically merges Stable Diffusion models by mixing specific layers (feedforward, attention, or all) based on prompt-derived gate weights. This approach allows for the creation of larger, more capable models on-the-fly by leveraging the distinct strengths of individual fine-tuned models, inspired by similar techniques in large language models.

Quick Start & Requirements

• Install via pip: pip install segmoe
• Requires CUDA-enabled GPU (e.g., 19GB for SDXL 2xN, 25GB for SDXL 4xN, 7GB for SD 1.5 4xN).
• Usage examples and detailed configuration guides are available in the official documentation.

Highlighted Details

• Enables creation of MoE models from Hugging Face or CivitAI links without training.
• Supports both Stable Diffusion 1.5 and SDXL models.
• Integrates with Diffusers pipelines for Image-to-Image and Inpainting tasks.
• Offers pre-released MoE models like SegMoE-2x1-v0 and SegMoE-4x2-v0.

Maintenance & Community

• Developed by Segmind.
• Roadmap includes optimizations for speed and memory, LoRA support, and more model integrations.

Licensing & Compatibility

• The repository does not explicitly state a license in the README. Compatibility for commercial use or closed-source linking is not specified.

Limitations & Caveats

The framework is not yet optimized for speed or memory usage. While it improves image fidelity and adherence, it does not surpass the performance of a single expert without further training.