MediaPipe4U is an Unreal Engine plugin enabling real-time AI/ML features like motion capture, facial expression tracking, text-to-speech, and speech recognition within UE projects. It targets game developers and 3D animators seeking to integrate advanced AI functionalities without external dependencies, offering a comprehensive toolkit for character animation and interactive experiences.

How It Works

The plugin leverages C++ integration with Google's MediaPipe framework and NVIDIA Maxine AR SDK for core motion and facial tracking. It incorporates various open-source models for Text-to-Speech (TTS) and Automatic Speech Recognition (ASR), including support for LLM inference via Ollama. This approach allows for offline, low-latency processing directly within Unreal Engine, supporting multiple input sources like webcams, video files, and Android devices.

Quick Start & Requirements

• Install: Download precompiled plugin binaries for specific UE versions (UE 5.5 currently available) from provided links. Copy plugin content to your project's Plugins folder.
• Prerequisites: Unreal Engine 5.5 (other versions planned). Windows and Android platforms are supported; Linux is in development. Manual OpenCV compilation may be required on first editor launch for UE 5.4+.
• Resources: Decompressed plugin requires ~10GB disk space; packaged projects use ~300MB.
• Docs: 中文文档, English (Google Translate)

Highlighted Details

• Full C++ implementation, no external Python or programs required.
• Supports LLM integration (Ollama, LLaMA, etc.), TTS (PaddleSpeech, Bark, F5-TTS), and ASR (FunASR, Whisper).
• Includes an open-source Android app (M4U Remoting) for facial capture via mobile devices.
• Recent updates add Ollama support, new TTS/ASR models, voice wake-up, and NvAR pose tracking.

Maintenance & Community

• Actively updated with frequent feature additions and bug fixes.
• Community support via GitHub Issues.
• Video tutorials available in English and Chinese.

Licensing & Compatibility

• The plugin uses a custom license. A free license file is available via GitHub discussions.
• Compatible with commercial use and closed-source linking, provided the license terms are met.

Limitations & Caveats

• The latest Google Holistic Task API does not support GPU inference, relying on CPU for Windows and Android.
• UE 5.4+ requires manual OpenCV compilation on first editor launch, which can be time-consuming and may require a VPN for downloads.
• Motion capture features have been moved to a separate plugin (MediaPipe4UMotion), potentially breaking existing Blueprints. Old licenses are invalidated.