Semantics Lead the Way (SFD) introduces a novel latent diffusion paradigm that harmonizes semantic and texture modeling for image generation. It addresses the limitation of synchronous denoising in existing Latent Diffusion Models (LDMs) by explicitly prioritizing semantic formation, enabling earlier semantic denoising to guide texture generation. This approach offers state-of-the-art FID scores and significantly accelerates training convergence, making it beneficial for researchers and practitioners in generative AI seeking high-quality, efficient image synthesis.

How It Works

SFD constructs composite latents by combining compact semantic representations from a pre-trained visual encoder with texture latents. It employs asynchronous denoising with separate noise schedules, allowing semantic latents to denoise first, establishing a semantic anchor. This is followed by a joint but asynchronous denoising phase where semantics lead textures, and finally, a texture completion phase. This explicit, semantics-led, coarse-to-fine generation process leverages the inherent structure of LDMs for improved quality and faster convergence.

Quick Start & Requirements

• Environment: Python 3.10.12, pip install -r requirements.txt, numpy==1.24.3, protobuf==3.20.0. Requires cloning and installing guided-diffusion (tensorflow==2.8.0).
• Model Weights: Download from Hugging Face: SD-VAE decoder (hustvl/va-vae-imagenet256-experimental-variants) and SFD models (SFD-Project/SFD).
• Data: ImageNet 256x256 batches.
• Hardware: Training and evaluation were performed on 16 NPUs; testing on 8 A100 GPUs showed minor variations.
• Links:
  • guided-diffusion: https://github.com/openai/guided-diffusion.git
  • Model Weights: https://huggingface.co/hustvl/va-vae-imagenet256-experimental-variants, https://huggingface.co/SFD-Project/SFD

Highlighted Details

• Achieves state-of-the-art FID scores, including 1.04 on ImageNet 256x256 with the 1.0B LightningDiT-XXL model.
• Demonstrates remarkable training convergence acceleration: approximately 100x faster than DiT and 33.3x faster than LightningDiT.
• Improves upon existing methods like ReDi and VA-VAE through its asynchronous, semantics-led modeling approach.
• Offers enhanced performance with an "AutoGuidance" module, achieving FID scores as low as 1.03 (SFD-XL, 800 epochs) and 1.04 (SFD-XXL, 800 epochs).

Maintenance & Community

The project's code is based on LightningDiT, REPA, and ADM repositories. No specific community channels (e.g., Discord, Slack), roadmap, or dedicated maintenance team beyond the listed authors are mentioned in the README.

Licensing & Compatibility

The README does not specify a license. As is common with research publications, it is likely intended for non-commercial, research-only use. Compatibility for commercial applications or linking with closed-source projects is not addressed.

Limitations & Caveats

The training code for the Semantic VAE and the main SFD diffusion model is currently listed as a to-do item and is not yet released. Performance results are primarily based on 16 NPU hardware, with minor discrepancies noted on A100 GPUs, suggesting potential hardware-specific tuning or precision differences.