Lyumin Zhang

Author of ControlNet

Authored Projects (8)

Starred by

Chip Huyen(Author of "AI Engineering", "Designing Machine Learning Systems"),

Jeff Hammerbacher(Cofounder of Cloudera),

Jesse Clark(Cofounder of Marqo),

Alex Yu(Research Scientist at OpenAI; Former Cofounder of Luma AI), and

1 more.

Fooocus by lllyasviel

0.1%

46k

Image generator for streamlined prompting and generation using SDXL

Simplified UI focuses on prompts/images, minimizing manual tweaking.
Employs offline GPT-2 for prompt processing, improving result quality.
Custom inpaint algorithm/models enhance inpainting beyond standard SDXL.
Supports multiple platforms (Windows/Linux/Mac) and AMD GPUs (beta).

Created 2 years ago

Updated 2 weeks ago

stable-diffusion-webui-forge by lllyasviel

0.2%

12k

Stable Diffusion platform for easier development, resource management, and faster inference

Based on Stable Diffusion WebUI, enhances development and experimentation.
Optimizes resource management and accelerates inference speeds.
Supports Flux BNB NF4 / GGUF with GPU weight slider and swap toggles.
Integrates FreeU V2 for improved image generation.

Created 1 year ago

Updated 1 month ago

Starred by

Yaowei Zheng(Author of LLaMA-Factory)

Dror Weiss(Cofounder of Tabnine)

Junyang Lin(Core Maintainer at Alibaba Qwen)

FramePack by lllyasviel

0.3%

16k

Desktop software for video generation via next-frame prediction

Compresses input contexts, enabling long video generation on laptop GPUs.
Supports PyTorch, xFormers, flash-attn, and sage-attention kernels.
Includes GUI for image upload, prompt input, and video preview.
Recommends RTX 30/40/50 series Nvidia GPUs with >=6GB memory.

Created 5 months ago

Updated 2 months ago

LuminaBrush by lllyasviel

776

Interactive tool for illumination effects on images

Converts images to uniformly-lit appearances.
Generates illumination effects via user scribbles.
Two-stage method simplifies learning, avoids constraints.
Uses Flux LoRA for generating training images.

Created 9 months ago

Updated 4 months ago

sd-forge-layerdiffuse by lllyasviel

0.0%

WebUI extension for transparent image layer diffusion

Generates transparent images/layers using latent transparency with SD WebUI (Forge).
Supports foreground/background conditioning for blended compositions via LoRA offsets.
Includes VAE encoder/decoder for latent offset extraction/image reconstruction.
Enables joint layer generation with attention sharing for improved structure understanding.

Created 1 year ago

Updated 1 year ago

Starred by

Didier Lopes(Founder of OpenBB) and

Patrick von Platen(Author of Hugging Face Diffusers; Research Engineer at Mistral).

ControlNet-v1-1-nightly by lllyasviel

ControlNet 1.1 is a research project for neural network control of diffusion models

Canny, MLSD, scribble, soft edge, segmentation, pose, lineart, shuffle, IP2P, inpaint, tile.
Enables control over Stable Diffusion using various input modalities.
Includes 14 models with improved robustness and result quality.
Supports multi-ControlNet setups via Automatic1111 plugin.

Created 2 years ago

Updated 1 year ago

Starred by

Elvis Saravia(Founder of DAIR.AI),

Pietro Schirano(Founder of MagicPath),

Charlie Holtz(Founder of Melty),

Jiaming Song(Chief Scientist at Luma AI), and

1 more.

Omost by lllyasviel

0.0%

Image composer using LLMs to generate code for image creation

Converts LLM coding to image composing via a virtual Canvas agent.
Pretrained LLM models based on Llama3 and Phi3 variations.
Uses direct preference optimization (DPO) for code compilation.
Includes a baseline renderer based on attention manipulation.

Created 1 year ago

Updated 1 year ago

Starred by

Vincent Weisser(Cofounder of Prime Intellect),

Benjamin Bolte(Cofounder of K-Scale Labs),

Louis Vichy(Cofounder of OpenRouter),

Gabriel Almeida(Cofounder of Langflow), and

26 more.

ControlNet by lllyasviel

0.1%

33k

Neural network structure for adding conditional control to diffusion models

Controls Stable Diffusion by adding extra conditions via a trainable copy.
Preserves model fidelity using a locked copy of network block weights.
Employs zero convolutions to avoid distortion before training.
Supports Canny edge, M-LSD lines, HED boundary, scribbles, pose, depth & segmentation.

Created 2 years ago

Updated 1 year ago