Lyumin Zhang

Author of ControlNet

Authored Projects (9)

VCC by lllyasviel

Compiler for agent conversation logs

Transforms raw logs into structured, searchable views for trace analysis.
Enables precise context recall and navigation across extensive chat histories.
Utilizes compiler stages (lex, parse, IR, emit) for robust log processing.
Generates dynamic views on-the-fly, unlike static memory systems.

Created 1 week ago

Updated 1 week ago

Starred by

Chip Huyen

Chip Huyen(Author of "AI Engineering", "Designing Machine Learning Systems"),

Jeff Hammerbacher

Jeff Hammerbacher(Cofounder of Cloudera),

Jesse Clark

Jesse Clark(Cofounder of Marqo),

Alex Yu

Alex Yu(Research Scientist at OpenAI; Cofounder of Luma AI), and

1 more.

Fooocus by lllyasviel

Image generator for streamlined prompting and generation using SDXL

Simplified UI focuses on prompts/images, minimizing manual tweaking.
Employs offline GPT-2 for prompt processing, improving result quality.
Custom inpaint algorithm/models enhance inpainting beyond standard SDXL.
Supports multiple platforms (Windows/Linux/Mac) and AMD GPUs (beta).

Created 2 years ago

Updated 4 months ago

Starred by

Yaowei Zheng

Yaowei Zheng(Author of LLaMA-Factory)

Dror Weiss

Dror Weiss(Cofounder of Tabnine)

Junyang Lin

Junyang Lin(Core Maintainer at Alibaba Qwen)

FramePack by lllyasviel

Desktop software for video generation via next-frame prediction

Compresses input contexts, enabling long video generation on laptop GPUs.
Supports PyTorch, xFormers, flash-attn, and sage-attention kernels.
Includes GUI for image upload, prompt input, and video preview.
Recommends RTX 30/40/50 series Nvidia GPUs with >=6GB memory.

Created 1 year ago

Updated 5 months ago

stable-diffusion-webui-forge by lllyasviel

Stable Diffusion platform for easier development, resource management, and faster inference

Based on Stable Diffusion WebUI, enhances development and experimentation.
Optimizes resource management and accelerates inference speeds.
Supports Flux BNB NF4 / GGUF with GPU weight slider and swap toggles.
Integrates FreeU V2 for improved image generation.

Created 2 years ago

Updated 8 months ago

LuminaBrush by lllyasviel

Interactive tool for illumination effects on images

Converts images to uniformly-lit appearances.
Generates illumination effects via user scribbles.
Two-stage method simplifies learning, avoids constraints.
Uses Flux LoRA for generating training images.

Created 1 year ago

Updated 11 months ago

sd-forge-layerdiffuse by lllyasviel

WebUI extension for transparent image layer diffusion

Generates transparent images/layers using latent transparency with SD WebUI (Forge).
Supports foreground/background conditioning for blended compositions via LoRA offsets.
Includes VAE encoder/decoder for latent offset extraction/image reconstruction.
Enables joint layer generation with attention sharing for improved structure understanding.

Created 2 years ago

Updated 1 year ago

Starred by

Didier Lopes

Didier Lopes(Founder of OpenBB) and

Patrick von Platen

Patrick von Platen(Author of Hugging Face Diffusers; Research Engineer at Mistral).

ControlNet-v1-1-nightly by lllyasviel

ControlNet 1.1 is a research project for neural network control of diffusion models

Canny, MLSD, scribble, soft edge, segmentation, pose, lineart, shuffle, IP2P, inpaint, tile.
Enables control over Stable Diffusion using various input modalities.
Includes 14 models with improved robustness and result quality.
Supports multi-ControlNet setups via Automatic1111 plugin.

Created 3 years ago

Updated 1 year ago

Starred by

Elvis Saravia

Elvis Saravia(Founder of DAIR.AI),

Pietro Schirano

Pietro Schirano(Founder of MagicPath),

Charlie Holtz

Charlie Holtz(Founder of Melty),

Jiaming Song

Jiaming Song(Chief Scientist at Luma AI), and

1 more.

Omost by lllyasviel

Image composer using LLMs to generate code for image creation

Converts LLM coding to image composing via a virtual Canvas agent.
Pretrained LLM models based on Llama3 and Phi3 variations.
Uses direct preference optimization (DPO) for code compilation.
Includes a baseline renderer based on attention manipulation.

Created 1 year ago

Updated 1 year ago

Starred by

Vincent Weisser

Vincent Weisser(Cofounder of Prime Intellect),

Benjamin Bolte

Benjamin Bolte(Cofounder of K-Scale Labs),

Louis Vichy

Louis Vichy(Cofounder of OpenRouter),

Gabriel Almeida

Gabriel Almeida(Cofounder of Langflow), and

26 more.

ControlNet by lllyasviel

Neural network structure for adding conditional control to diffusion models

Controls Stable Diffusion by adding extra conditions via a trainable copy.
Preserves model fidelity using a locked copy of network block weights.
Employs zero convolutions to avoid distortion before training.
Supports Canny edge, M-LSD lines, HED boundary, scribbles, pose, depth & segmentation.

Created 3 years ago

Updated 2 years ago

Starred Projects (8)

Fooocus by lllyasviel

Image generator for streamlined prompting and generation using SDXL

chiphuyen:

hammer:

jn2clark:

sxyu:

Created 2 years ago

Updated 4 months ago

docker-prompt-generator by soulteary

Docker image for prompt generation

omarsar:

Created 3 years ago

Updated 3 years ago

ControlNet by lllyasviel

Neural network structure for adding conditional control to diffusion models

vincentweisser:

codekansas:

louisgv:

ogabrielluiz:

Created 3 years ago

Updated 2 years ago

T2I-Adapter by TencentARC

T2I-Adapter for controllable text-to-image diffusion models (SD-XL)

sxyu:

chenlin9:

osanseviero:

patrickvonplaten:

Created 3 years ago

Updated 1 year ago

kohya-trainer by Linaqruf

Trainer for Stable Diffusion models, adapted for easier use

Created 3 years ago

Updated 1 year ago

stable-diffusion-webui by AUTOMATIC1111

Web UI for Stable Diffusion

clmnt:

jmorganca:

k06a:

vincentweisser:

Created 3 years ago

Updated 1 month ago

manga-image-translator by zyddnys

Image translator for manga/images, supporting multiple languages

Created 5 years ago

Updated 2 weeks ago

Deep-reinforcement-learning-with-pytorch by sweetice

PyTorch library for deep reinforcement learning algorithms

shizhediao:

evhub:

Created 7 years ago

Updated 3 years ago

Feedback? Help us improve.