FreeNoise enables tuning-free, longer video generation using pre-trained diffusion models, targeting researchers and developers in AI video synthesis. It offers a significant benefit by extending video length with minimal computational overhead and no fine-tuning required.

How It Works

FreeNoise implements a novel "noise rescheduling" technique. Instead of standard diffusion sampling, it strategically modifies the noise schedule during inference. This allows the model to generate longer sequences by effectively "rescheduling" the noise levels across more timesteps, maintaining coherence and quality without altering the original model weights.

Quick Start & Requirements

• Install: conda create -n freenoise python=3.8.5, conda activate freenoise, pip install -r requirements.txt.
• Prerequisites: Pretrained T2V models (e.g., VideoCrafter) must be downloaded and placed in the checkpoints/ directory.
• Resources: Requires NVIDIA A100 (40GB) for 512 frames at 256x256 or 64 frames at 1024x576. Smaller GPUs may require reducing frame counts.
• Links: Hugging Face Spaces, Replicate

Highlighted Details

• Tuning-free longer video generation.
• Less than 20% extra generation time.
• Supports up to 512 frames.
• Compatible with VideoCrafter, AnimateDiff, and LaVie frameworks.

Maintenance & Community

The project is associated with Tencent AI Lab and Nanyang Technological University. Key contributors include Haonan Qiu, Menghan Xia, Yong Zhang, Yingqing He, Xintao Wang, Ying Shan, and Ziwei Liu.

Licensing & Compatibility

The repository is developed for RESEARCH purposes and is restricted to personal, research, and non-commercial use.

Limitations & Caveats

The primary limitation is the non-commercial use restriction due to its research-focused nature. Specific hardware requirements (e.g., NVIDIA A100) are noted for achieving maximum frame counts.