MM-StoryAgent is a multi-agent framework for generating immersive narrated storybook videos. It leverages Large Language Models (LLMs) and specialized tools across text, image, and audio modalities to create expressive storytelling content, offering a customizable workflow for users to integrate their own expert agents.

How It Works

MM-StoryAgent employs a multi-agent, multi-stage pipeline to generate stories and corresponding assets. LLMs are used to write high-quality stories based on user-defined settings. Separate agents handle the generation of image, speech, sound, and music assets, which are then composed into a final video. This modular, agent-based approach allows for customization and improved generation quality for each component.

Quick Start & Requirements

• Install dependencies and the package:

  pip install -r requirements.txt
  pip install -e .
  

• Run generation via configuration files:

  python run.py -c configs/mm_story_agent.yaml
  

• Custom agents can be added by implementing __init__ and call methods and registering them.

Highlighted Details

• Generates expressive storytelling videos by composing text, image, speech, sound, and music assets.
• Features a customizable workflow allowing users to define and integrate their own expert tools.
• Employs a multi-agent, multi-stage pipeline for story writing, outperforming direct LLM prompting in evaluations.
• Evaluation data, rubrics, and prompts are provided for story quality assessment.

Maintenance & Community

• The initial version was released on August 16, 2024.
• A demo video is available.

Licensing & Compatibility

• The repository does not explicitly state a license in the provided README.

Limitations & Caveats

• The README does not specify hardware requirements (e.g., GPU) or detailed setup time.
• No explicit mention of supported operating systems or specific Python versions beyond general dependency installation.