BiliSum is an AI-powered tool for creating searchable knowledge bases from Bilibili, YouTube, and local videos. It targets users who need to efficiently extract, organize, and query information from video content, transforming raw video data into structured, retrievable knowledge assets.

How It Works

The core pipeline processes video content through automatic speech recognition (ASR) to generate transcriptions, which are then used to create structured text notes. A novel "VLM Understanding" feature leverages multimodal large language models (VLMs) to interpret video frames, generating image-rich notes where paragraphs are intelligently interspersed with relevant screenshots. Summarization identifies key arguments and conclusions, while a Retrieval-Augmented Generation (RAG) knowledge base enables cross-video semantic search and querying. Users can opt for various ASR engines, including fast local options like FunASR (QwenASR) and SiliconFlow, alongside Whisper.

Quick Start & Requirements

Installation involves uv sync --python 3.12 --all-packages for Python dependencies and npm install --prefix ./apps/desktop for the Node.js frontend. The browser version can be launched with npx bilisum. A desktop application is available for Windows and macOS. Prerequisites include Python 3.12+ and Node.js 20+. Optional dependencies like ffmpeg and CUDA are recommended for certain functionalities. Docker images are also provided for deployment. Links to the quick start guide are available within the README.

Highlighted Details

• VLM Understanding Notes: Generates structured notes by interleaving LLM-processed paragraphs with VLM-interpreted screenshots, offering a novel approach to visual content integration.
• Multi-Provider LLM/VLM Support: Seamlessly integrates with OpenAI, Anthropic, and compatible APIs for both text and vision models, with automatic configuration.
• Flexible ASR Options: Supports SiliconFlow ASR, FunASR (QwenASR) for high-performance local Chinese transcription, multimodal ASR, and local Whisper.
• RAG Knowledge Base: Enables cross-video semantic search, keyword querying, tag management, and visualization, with support for local LLMs for offline use.
• Bilibili Integration: Features built-in QR code scanning for Bilibili login and automatic cookie management to bypass anti-scraping measures.
• Cross-Platform Desktop App: A native desktop application built with Electron is available for Windows and macOS.

Maintenance & Community

The project is maintained by Lycohana. A detailed contribution guide is provided, outlining bug reporting and pull request processes. The roadmap indicates ongoing development with planned features like Notion integration and broader platform support. Community links (e.g., Discord/Slack) are not explicitly listed.

Licensing & Compatibility

BiliSum is released under the MIT License, which permits commercial use and integration into closed-source projects without significant restrictions.

Limitations & Caveats

The VLM Understanding feature incurs higher API costs and is still under active optimization, with potential for simplified notes on very long videos. While a desktop app exists for Windows and macOS, support for other platforms is listed as a future roadmap item. The project's roadmap indicates that some features are still under development.