Summary

JAVEdit addresses the lack of resources for instruction-guided joint audio-visual video editing. It provides JAVEdit-100k, the first large-scale dataset for this task, an automated agent-in-the-loop data curation pipeline, and a human-aligned benchmark (JAVEditBench). This project enables development and evaluation of advanced video editing models that synchronize visual and auditory elements via natural language instructions.

How It Works

JAVEdit-100k is a dataset with ~103K editing triplets (Subject/Background Editing, Removal, Addition, Speech), featuring free-form instructions and high-res video (1280x720, 121 frames, 25 FPS). Agent-in-the-loop quality control (Inspector/Orchestrator agents) automates failure detection, diagnosis, and repair, boosting data qualification from 36% to 83%. JAVEditBench provides a human-aligned evaluation suite (Spearman's ρ ≥ 0.80) on 150 curated videos, assessing visual-audio quality, instruction compliance, and video fidelity. The JAVEdit baseline model shows superior performance.

Quick Start & Requirements

Install: Clone repo, setup Conda (conda create -n javeditbench python=3.12, conda activate javeditbench), pip install -r requirements.txt. Prerequisites: ffmpeg/ffprobe on PATH, CUDA driver compatible with CUDA 12.8 for PyTorch