Bailando is a framework for generating 3D dance sequences from music, targeting researchers and developers in computer vision and animation. It addresses the challenge of creating synchronized, temporally coherent, and spatially constrained dances by leveraging a choreographic memory and an actor-critic GPT model.

How It Works

The core of Bailando is a two-part system: a choreographic memory that quantifies dance units into a codebook, and an actor-critic GPT that composes these units. This approach allows dance generation to operate on quantized units that adhere to choreography standards, ensuring spatial constraints are met. An actor-critic reinforcement learning scheme with a beat-align reward function synchronizes motion tempos with music beats.

Quick Start & Requirements

• Install: Requires PyTorch 1.6.0.
• Data: Uses the AIST++ dataset. Requires downloading annotations, music, SMPL models, and running ./prepare_aistpp_data.sh. Preprocessed features are available.
• Hardware: Models are trained on a single NVIDIA V100 GPU. Quantization does not fit multi-GPU training.
• Links: Paper, Project Page, Video Demo

Highlighted Details

• State-of-the-art performance on standard benchmarks, both qualitatively and quantitatively.
• Choreographic memory discovers human-interpretable dancing poses unsupervised.
• Actor-critic finetuning enables generation for "music in the wild."

Maintenance & Community

• The project is associated with the CVPR 2022 paper "Bailando" and an updated TPAMI 2023 version, "Bailando++."

Licensing & Compatibility

• Licensed under NTU S-Lab License 1.0. Redistribution and use must follow this license.

Limitations & Caveats

The reinforcement learning finetuning process for "music in the wild" is not guaranteed to produce satisfying results, and empirical success is observed after <= 30 epochs. The original AIST++ dataset may not cover all music genres or complexities found in real-world dance music.