FramePack  by lllyasviel

Desktop software for video generation via next-frame prediction

created 3 months ago
15,336 stars

Top 3.3% on sourcepulse

GitHubView on GitHub
Project Summary

FramePack offers a novel approach to video generation by treating it as a next-frame prediction task, enabling efficient, progressive video creation. It compresses input contexts to a fixed length, making generation workload independent of video duration, and allows for large batch sizes akin to image diffusion training. This makes it accessible for users with limited hardware, including laptop GPUs, to generate longer videos.

How It Works

FramePack employs a next-frame prediction neural network architecture that generates videos sequentially. Its core innovation lies in compressing input contexts into a constant-length representation. This design choice decouples generation complexity from video length, allowing for efficient processing of extended sequences and enabling training with larger batch sizes, similar to image diffusion models.

Quick Start & Requirements

  • Windows: Download the one-click package (CUDA 12.6 + PyTorch 2.6). Uncompress, run update.bat, then run.bat. Models download automatically (30GB+).
  • Linux: Install PyTorch with CUDA 12.6, then pip install -r requirements.txt. Run GUI with python demo_gradio.py.
  • Requirements: NVIDIA GPU (RTX 30XX, 40XX, 50XX series supporting fp16/bf16), 6GB+ VRAM, Linux/Windows. GTX 10XX/20XX not tested.
  • Resources: Expect 30GB+ model downloads. Generation speed varies by GPU (e.g., 1.5s/frame on RTX 4090, 4-8x slower on laptop GPUs).
  • Links: Paper, Project Page

Highlighted Details

  • Generates 1-minute videos (1800 frames) at 30fps with 6GB VRAM using 13B models.
  • Provides real-time visual feedback as frames are generated.
  • Supports various attention mechanisms (PyTorch, xformers, flash-attn, sage-attention).
  • Warns that "teacache" and other optimizations can influence results.

Maintenance & Community

The project is associated with researchers from Meta AI and Princeton University. The team is on leave from April 21-30, which may delay PR merging.

Licensing & Compatibility

The repository does not explicitly state a license. The paper is available on arXiv.

Limitations & Caveats

The project is described as a functional desktop software with a minimal standalone sampling system. Users are advised that results can be sensitive to hardware and software configurations, and optimizations like "teacache" may impact output quality. The project also warns against numerous unofficial websites claiming to offer FramePack services.

Health Check
Last commit

2 weeks ago

Responsiveness

1 day

Pull Requests (30d)
2
Issues (30d)
22
Star History
3,849 stars in the last 90 days

Explore Similar Projects

Feedback? Help us improve.