Summary

This project provides a standalone C++ inference engine for 3D full-body reconstruction from a single camera feed. It targets engineers and researchers needing efficient, real-time human pose estimation without runtime Python dependencies. The engine leverages ONNX Runtime and ggml to output multi-person BVH motion capture files, a 70-joint skeleton including hands, and optionally full 3D mesh vertices, enabling direct integration into 3D workflows like Blender.

How It Works

The system directly regresses 3D body model parameters from image features, bypassing traditional 2D-to-3D lifting. The pipeline begins with YOLO for person detection, followed by a DINOv3-ViT-H backbone for feature extraction. A transformer decoder compresses these features into a pose token, which is then decoded by lightweight heads (run via ggml) into MHR pose parameters and camera translation. Linear Blend Skinning (LBS) is applied using native C data to generate the full 3D mesh and keypoints. This approach avoids the need for depth sensors or stereo vision.

Quick Start & Requirements

• Installation: Requires CMake >= 3.18 and a C++17 compiler. Build using cmake .. -DCMAKE_BUILD_TYPE=Release within a build directory, followed by make -j$(nproc). OpenCV is also a dependency.
• Models: Pre-built ONNX/GGUF models must be downloaded from HuggingFace (https://huggingface.co/AmmarkoV/SAM3DBody-cpp-onnx-models). The CUDA version requires ~4.8 GB, while the CPU-compatible backbone adds ~3.2 GB.
• Hardware: A CUDA-capable GPU is strongly recommended for real-time performance. CPU-only inference is impractically slow for video.

Highlighted Details

• Real-time 3D full-body reconstruction from a single camera.
• Pure C++ runtime with zero Python dependency.
• Multi-person BVH motion-capture export, with identity tracking across frames.
• Outputs a 70-joint skeleton, including hands and face expression parameters.
• Integrates with Blender via a provided plugin for MakeHuman-rigged characters.
• Supports ONNX Runtime and ggml for efficient inference.

Maintenance & Community

The project is authored by Ammar Qammaz. No specific community channels (e.g., Discord, Slack) or explicit maintenance signals beyond the repository itself are detailed in the README.

Licensing & Compatibility

The provided README does not explicitly state the software license. This omission requires further investigation for commercial use or integration into closed-source projects.

Limitations & Caveats

CPU-only inference is significantly slower than GPU, making video processing impractical. Model files are large, requiring substantial download and storage. The README does not specify a software license, which is a critical factor for adoption.