Summary

This project provides a C++17 inference engine for NVIDIA's LocateAnything-3B model, built on the ggml library. It enables fast, dependency-light, open-vocabulary object detection from text prompts, primarily targeting CPU execution while supporting GPU acceleration. The key benefit is significantly faster inference speeds and reduced resource requirements compared to Python-based implementations, without compromising detection accuracy.

How It Works

The engine integrates Qwen2.5-3B (LM) and MoonViT (vision) models, using a 2-layer MLP projector. Detection is performed in token space, where coordinate tokens are emitted and decoded into bounding boxes. This approach, implemented in C++ with ggml, allows for efficient execution on commodity hardware, bypassing the need for a Python runtime during inference.

Quick Start & Requirements

• Primary Install/Run: Clone the repository with submodules (git clone --recursive), then build using CMake (cmake -B build -DLA_BUILD_CLI=ON && cmake --build build -j). GPU support requires -DLA_GGML_CUDA=ON.
• Prerequisites: C++17 compiler, CMake. Python is needed for model conversion scripts. Pre-built GGUF models are available on Hugging Face (mudler/locate-anything.cpp-gguf).
• Links: Repository: https://github.com/mudler/locate-anything.cpp

Highlighted Details

• Achieves identical detections to the official LocateAnything-3B, verified through rigorous benchmarking.
• Offers substantial speedups on CPU: ~1.7-3.1x faster than PyTorch-CPU implementations for various modes.
• Quantized GGUF models (e.g., q8_0 at ~6.3 GB) provide near-lossless accuracy with further performance gains.
• GPU acceleration (CUDA) provides additional speedups, outperforming official bf16 models.

Maintenance & Community

Developed by the LocalAI team, with contributions noted from Ettore Di Giacinto and Richard Palethorpe. No specific community channels (Discord, Slack) or roadmap links were found in the provided README.

Licensing & Compatibility

The C++ code is licensed under MIT, permitting commercial use and integration into closed-source projects. The underlying model weights are subject to NVIDIA's separate license.

Limitations & Caveats

The implementation intentionally excludes stochastic sampling methods (temperature, top_p) as they degrade detection accuracy; inference is strictly greedy. Multi-image prompts are not supported due to limitations in output attribution, making it a single-image detection engine.