VBench provides a comprehensive benchmark suite for evaluating video generative models, targeting researchers and developers in the field of AI video generation. It offers a structured framework to assess various quality dimensions, enabling fine-grained and objective comparisons between different models.

How It Works

VBench decomposes "video generation quality" into 16 well-defined dimensions, each with a specific prompt suite and an automated evaluation method. It supports both Text-to-Video (T2V) and Image-to-Video (I2V) tasks, and can evaluate custom videos. The framework also incorporates human preference annotations to ensure alignment with human perception, and recent updates (VBench-2.0) extend evaluation to intrinsic faithfulness aspects like commonsense reasoning and physics.

Quick Start & Requirements

• Installation: pip install vbench (requires PyTorch with CUDA <= 12.1). detectron2 is needed for some evaluations (pip install detectron2@git+https://github.com/facebookresearch/detectron2.git), which requires CUDA 11.X or 12.1.
• Data: Download VBench_full_info.json for prompt suites.
• Usage: vbench evaluate --videos_path <path> --dimension <dimension> or via Python API.
• Links: Leaderboard, Model Info, Prompt Suites

Highlighted Details

• Comprehensive evaluation across 16 dimensions, including technical quality and trustworthiness.
• Supports both T2V and I2V models, with extensions for longer videos (VBench-Long) and intrinsic faithfulness (VBench-2.0).
• VBench Arena allows users to view and vote on generated videos from over 40 supported models.
• Released sampled videos and detailed model settings for transparency and reproducibility.

Maintenance & Community

• Actively maintained with frequent updates, including VBench-2.0 and human anomaly detection pipelines.
• Community engagement via GitHub issues and a Google Form for evaluation requests.
• Related project: Awesome-Evaluation-of-Visual-Generation.

Licensing & Compatibility

• The repository itself is likely under a permissive license (e.g., MIT, Apache 2.0, based on common practice for such projects), but specific license details are not explicitly stated in the README.
• Compatibility for commercial use is generally expected for permissive licenses, but users should verify the specific license.

Limitations & Caveats

• Detectron2 installation can be problematic and is restricted to specific CUDA versions (11.X or 12.1).
• Some evaluation dimensions require specific dependencies or preprocessing steps (e.g., static video filtering for temporal flickering).