This project provides inference optimization for ComfyUI, targeting users seeking faster image and video generation. It offers universal, flexible, and fast solutions through dynamic caching and enhanced torch.compile integration, aiming to significantly reduce computation costs and generation times.

How It Works

The core of the optimization lies in two main techniques. "First Block Cache" (FBCache) leverages the residual output of the initial transformer block. If subsequent residual outputs are similar to the previous ones, it reuses cached results, skipping later computations for up to 2x speedup. "Enhanced torch.compile" compiles model components for faster execution, notably supporting LoRA models, unlike the original TorchCompileModel node.

Quick Start & Requirements

• Install via git clone into ComfyUI's custom_nodes directory.
• Requires ComfyUI.
• torch.compile node has specific software/hardware requirements; refer to the Enhanced torch.compile section. FP8 quantization with torch.compile is not supported on pre-Ada GPUs (e.g., RTX 3090). torch.compile is not officially supported on Windows.
• Demo workflows are available in the workflows folder.

Highlighted Details

• First Block Cache (FBCache) offers 1.5x to 3.0x speedup with acceptable accuracy loss.
• Supports various models including FLUX, LTXV, HunyuanVideo, SD3.5, and SDXL.
• Enhanced torch.compile node works with LoRA models.
• FBCache is incompatible with the FreeU Advanced node pack for SDXL.

Maintenance & Community

• Project is marked as [WIP] (Work In Progress).
• Users are encouraged to join the Discord server for requests and questions.

Licensing & Compatibility

• License is not explicitly stated in the README.

Limitations & Caveats

• Multi-GPU inference is listed as a future feature ([WIP]).
• torch.compile may have issues with model offloading and requires specific configurations for optimal performance and to avoid recompilation.
• FP8 quantization with torch.compile is not supported on older GPUs.
• torch.compile is not officially supported on Windows.