CLIPasso offers a method for converting images into abstract sketches using Bézier curves, controlled by semantic understanding from CLIP. It's designed for researchers and artists interested in generative art and image abstraction, providing a novel approach to sketch synthesis.

How It Works

CLIPasso defines a sketch as a set of Bézier curves, optimizing their parameters via a differentiable rasterizer (diffvg). This optimization is guided by a CLIP-based perceptual loss, leveraging both intermediate and final activations of a pre-trained CLIP model. This dual-activation approach allows for simultaneous geometric and semantic simplification, with the level of abstraction directly tunable by the number of strokes.

Quick Start & Requirements

• Recommended Installation: Docker (docker pull yaelvinker/clipasso_docker).
• Pip Installation: Requires Python 3.7, PyTorch 1.7.1+cu101, torchvision 0.8.2+cu101, and the CLIP library. Also requires manual installation of diffvg (which has specific compilation requirements).
• Demo: Run python run_object_sketching.py --target_file <file_name>.
• Resources: GPU is highly recommended due to potential slowness on CPU.

Highlighted Details

• Semantically-aware sketch generation using CLIP.
• Abstraction level controlled by the number of Bézier strokes.
• Output sketches in SVG format.
• Official implementation of SIGGRAPH 2022 paper.

Maintenance & Community

No specific community links (Discord/Slack) or roadmap are provided in the README. The project is associated with the SIGGRAPH 2022 publication.

Licensing & Compatibility

Licensed under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0). This license prohibits commercial use and requires derivative works to be shared under the same license.

Limitations & Caveats

The pip installation is noted as potentially problematic due to diffvg compilation issues. The CC BY-NC-SA 4.0 license restricts commercial applications. Running on CPU is not recommended due to significant performance degradation.