ClipSketch AI is a productivity tool designed to transform video moments into hand-drawn storyboards and generate engaging social media copy. It targets video creators, social media operators, and remix enthusiasts, offering an AI-driven workflow to streamline content creation and enhance engagement. The tool parses video links from platforms like Bilibili and Xiaohongshu, allowing users to precisely mark keyframes and then leverage Google Gemini's multimodal capabilities to generate stylized storyboards and tailored social media captions.

How It Works

The project employs a React/TypeScript frontend with Tailwind CSS for styling. It ingests video content via shared links from Bilibili and Xiaohongshu, providing a high-definition player with frame-level scrubbing and keyboard controls. Users can mark specific moments, which are then processed by Google's Gemini API. Specifically, gemini-3-pro-image-preview is used to synthesize marked frames into a cohesive, hand-drawn style storyboard, while gemini-3-pro-preview generates multiple styles of social media copy. Advanced features include custom character integration into scenes and AI-generated video cover art.

Quick Start & Requirements

• Primary install/run command: Clone the repository (git clone https://github.com/RanFeng/clipsketch-ai.git), navigate into the directory (cd clipsketch-ai), install dependencies (npm install), and start the development server (npm run dev).
• Non-default prerequisites: Node.js (v18+), a valid Google Gemini API Key.
• Access: The application is accessible at http://localhost:3000.
• Links: Official quick-start is within the README; Gemini API documentation is available via Google AI.

Highlighted Details

• Supports importing videos from Bilibili and Xiaohongshu via share links.
• Features a frame-level marking system with keyboard shortcuts (Space, Left/Right arrows, T key).
• AI-powered generation of hand-drawn style storyboards and social media copy (3 styles).
• Enables custom character integration into generated storyboards.
• Provides AI-generated video cover creation based on selected copy and visuals.
• Offers batch processing for storyboard refinement and cost optimization.
• Responsive design ensures compatibility across PC, tablet, and mobile devices.

Maintenance & Community

No specific details regarding maintainers, community channels (like Discord/Slack), or roadmap were provided in the README snippet.

Licensing & Compatibility

• License type: MIT License.
• Compatibility: The MIT license is permissive, generally allowing for commercial use and integration into closed-source projects without significant restrictions.

Limitations & Caveats

Usage of AI features requires a Google Gemini API key with access to gemini-3-pro-image-preview and gemini-3-pro-preview models; incorrect API key configurations or permissions may result in 403 errors. The project utilizes specific proxy strategies and referrerPolicy="no-referrer" for handling external video playback and screenshots, which may impact functionality or security in certain environments.