MiniMates is a lightweight digital human driver algorithm designed for real-time performance on consumer hardware, including CPUs. It enables users to create personalized AI companions driven by voice or facial expressions, offering a significantly faster alternative to existing solutions like liveportrait and EchoMimic.

How It Works

MiniMates employs a coarse-to-fine wrap network architecture, eschewing traditional dense motion methods for improved CPU performance. It also utilizes explicit UV map techniques to enhance portrait accuracy. This approach allows for real-time inference on a wide range of devices, from integrated graphics to high-end GPUs.

Quick Start & Requirements

• Install:

  conda create -n MiniMates python=3.12
  conda activate MiniMates
  pip install torch --index-url https://download.pytorch.org/whl/cu124
  pip install -r requirements.txt
  

• Prerequisites: Python 3.12, PyTorch with CUDA 12.4 (for GPU acceleration).
• Usage: Pre-trained models are available via Baidu Netdisk (link and password provided). Examples for image matting, real-time face tracking, audio-driven generation, and template generation are included.
• Demo: release video

Highlighted Details

• Claims to be the fastest open-source digital human driver, supporting real-time inference on integrated graphics and CPUs.
• Features "one-shot" single-image driving for personalization.
• Offers voice and facial expression-driven modes.
• Recent updates include real-time dialogue and camera-based ARKit expression capture.

Maintenance & Community

• Active development with recent updates in October 2024.
• Community support via QQ and WeChat groups.

Licensing & Compatibility

• MIT License.
• Permissive license suitable for commercial use and integration into closed-source projects.

Limitations & Caveats

• The README notes that current expression driving is "not perfect" and may yield unstable but interesting results.
• A "To Do List" indicates features like desktop software integration and training code release are planned.