MiKaPo is a web-based application that enables real-time motion capture for MikuMikuDance (MMD) models using video input. It targets MMD users and creators looking for an accessible, browser-based solution to animate their models from video or webcam feeds. The primary benefit is the ability to quickly generate motion data for MMD models without complex software or hardware setups.

How It Works

The system leverages MediaPipe for 3D pose, face, and hand detection from video streams. Detected keypoints are then processed by a Rust-WASM solver to convert them into quaternions, facilitating 3D rotations. This processed data is fed into a Babylon.js-based MMD model viewer, allowing for real-time animation of MMD models within the browser. The use of WebAssembly for the solver aims to provide efficient, near-real-time performance.

Quick Start & Requirements

• Primary install / run command: npm install followed by npm run dev for development.
• Prerequisites: A modern web browser with access to a dedicated GPU is recommended for optimal performance.
• Links: Project Setup

Highlighted Details

• Real-time 3D pose, face, and hand detection via MediaPipe.
• Rust-WASM based pose-to-quaternion solver for efficient rotation calculation.
• Integrates with Babylon.js for MMD model rendering and manipulation.
• Supports video/image uploads and webcam input.
• VMD import/export functionality is available, requiring at least one recorded motion to export a valid VMD file.

Maintenance & Community

The project welcomes feature requests and pull requests. Community engagement channels are not explicitly listed in the README.

Licensing & Compatibility

The README does not specify a license. Compatibility for commercial use or closed-source linking is not detailed.

Limitations & Caveats

Hand detection is noted as experimental. The VMD export feature requires a motion to be recorded to produce a valid file. An Electron version with Ollama support is mentioned, but details on its setup and functionality are not provided in the README.