JoyAI-Echo addresses the persistent challenges in long-form video generation, such as error accumulation, temporal incoherence, and high latency, by introducing a novel framework for minute-level, multi-shot audio-visual content creation. It targets researchers and power users seeking to push the boundaries of generative AI for video, offering a significant leap in consistency and interactivity for extended narratives. The primary benefit is the ability to generate coherent, high-fidelity, and conversationally editable videos up to five minutes long.

How It Works

JoyAI-Echo employs a cross-modal audio-visual memory bank to maintain character appearance and voice timbre consistency across extended sequences. This memory is coupled with a post-training pipeline that integrates memory-based reinforcement learning and distribution matching distillation (DMD). This approach achieves a substantial 7.5x speedup over prior methods, enhancing visual quality and temporal alignment. The framework also incorporates an interactive agent for real-time editing via conversational commands and a lightweight super-resolution module for high-definition streaming output.

Quick Start & Requirements

Installation involves cloning the repository, followed by environment setup using either Conda (conda env create -f environment.yml) or uv (uv venv --python 3.11 .venv, source .venv/bin/activate, uv pip install --extra-index-url https://download.pytorch.org/whl/cu128 -r requirements.txt). A system-wide ffmpeg installation is required for shot concatenation. The recommended environment specifies Python 3.11, PyTorch 2.8, and CUDA 12.8. Model checkpoints, including the main model (~46 GB) and the Gemma text encoder (~24 GB), must be downloaded separately and placed in the checkpoints/ directory.

Highlighted Details

• Generates minute-level multi-shot stories (up to 5 minutes) from single prompt JSON files.
• Achieves approximately 7.5x faster inference through DMD-distilled few-step generation.
• Produces synchronized video and audio within a single pipeline.
• Utilizes a paired cross-modal memory bank for robust story-level consistency in identity and voice.
• Human evaluations show strong user preference for JoyAI-Echo over HappyOyster for long video aspects like visual aesthetics, audio quality, prompt following, and IP consistency.

Maintenance & Community

The project acknowledges contributions from LTX2.3 and Gemma. While specific community channels like Discord or Slack are not detailed, the project is hosted on GitHub and Hugging Face, indicating a public development presence. A TODO list mentions future releases for Echo-SR (Super-resolution) and a Director Agent, suggesting ongoing development.

Licensing & Compatibility

JoyAI-Echo is released under the LTX-2 Community License Agreement. It is explicitly stated that the project is for academic research and non-commercial use only and is not intended for commercial applications. Original LTX-2 copyright and license notices are retained.

Limitations & Caveats

The framework requires significant GPU memory, with peak usage around 46–50 GB, necessitating high-end hardware like an 80 GB H100/A100 or a 48 GB GPU. For systems with less VRAM, users must reduce video resolution or frame count. Features such as Echo-SR and the Director Agent are still pending release, indicating potential future enhancements or currently unavailable functionalities. Commercial use is strictly prohibited.