FlipSketch enables the creation of animated sketch sequences from static drawings, guided by text prompts. This project targets researchers and artists interested in generative AI for animation, offering a novel approach to transforming static visual concepts into dynamic, text-controlled visual narratives.

How It Works

The core of FlipSketch utilizes a Text-to-Video (T2V) model that has been specifically fine-tuned on sketch animation datasets. This fine-tuned model is then conditioned to adhere to an input sketch. The process involves attention composition, where reference noise derived from the input sketch is integrated to guide the animation generation, ensuring the output aligns with both the visual input and textual descriptions.

Quick Start & Requirements

• Install: Create a conda environment using conda env create -f flipsketch.yml.
• Model: Download the T2V LoRA model from HuggingFace (git lfs install git clone https://huggingface.co/Hmrishav/t2v_sketch-lora) and place the checkpoint in the root folder.
• Run: Execute the application with python app.py.
• Prerequisites: PyTorch 2+ is supported (requires modification to import from text2vid_torch2.py).
• Demo: A HuggingFace Space demo is available.

Highlighted Details

• Leverages a T2V model fine-tuned on sketch animations.
• Employs attention composition with reference noise from input sketches.
• Supports PyTorch 2+.
• Includes a HuggingFace Space for interactive demonstration.

Maintenance & Community

The project is associated with Hmrishav Bandyopadhyay and Yi-Zhe Song, with a CVPR 2025 publication. Further details and potential community interaction points are not explicitly listed in the README.

Licensing & Compatibility

The README does not specify a license. Compatibility for commercial use or closed-source linking is not detailed.

Limitations & Caveats

The project is presented as part of a CVPR 2025 submission, suggesting it may be in an early research phase. Specific hardware requirements beyond standard PyTorch dependencies are not detailed, and the exact nature of the "fine-tuned on sketch animations" dataset is not provided.