This project provides a local solution for enabling Large Language Models (LLMs) like Claude, ChatGPT, and Gemini to "watch" videos. It addresses the limitation of current AI tools that often rely solely on transcripts or sample frames at fixed, inefficient intervals. By processing videos locally, it extracts scene-aware, deduplicated frames and transcripts, offering a more comprehensive and cost-effective input for LLMs, suitable for researchers and power users analyzing video content.

How It Works

The tool fetches video content from URLs using yt-dlp or directly from local files. It employs ffmpeg for intelligent frame extraction, utilizing scene-change detection and a density floor to capture critical visual moments across both fast-cut sequences and static scenes. A sliding-window deduplication algorithm, based on pixel differences, ensures that visually similar frames or repeated shots are sent only once, optimizing context. Audio is transcribed using Whisper (or existing subtitles are leveraged), and all processed data—frames, transcript, and a MANIFEST.txt—is outputted locally for LLM consumption.

Quick Start & Requirements

• Installation:
  • Core: pip install claude-real-video
  • With transcription: pip install "claude-real-video[whisper]"
• System Prerequisites: ffmpeg and ffprobe must be installed and accessible in the system's PATH. Python 3.10+ is required.
• Usage: Execute via command line (crv "URL or file") or programmatically using from claude_real_video import process.
• Documentation: Primary details are available in the project's README.

Highlighted Details

• Intelligent frame selection via scene-change detection (--scene) and fps-floor parameter.
• Advanced deduplication (--dedup-threshold, --dedup-window) using pixel comparison to avoid redundant frames.
• Flexible input support for video URLs (e.g., YouTube, Instagram) and local files.
• Optional audio transcription via Whisper or fallback to existing subtitles, with an option to retain the full audio track (--keep-audio).
• All processing occurs locally, ensuring data privacy and avoiding cloud uploads.

Maintenance & Community

Information regarding maintainers, community channels (like Discord/Slack), or a public roadmap is not detailed in the provided README.

Licensing & Compatibility

The project is released under the MIT license, which is permissive and generally compatible with commercial use and integration into closed-source projects.

Limitations & Caveats

Users must ensure they possess the necessary rights to download and process video content. The ffmpeg dependency must be manually installed and configured in the system's PATH. The --cookies option requires careful management of credentials. Re-running the tool on an existing output directory will overwrite previous results.