This project provides an official, fully functional web demo for the MiniCPM-o 4.5 model, showcasing its advanced audio-video omnimodal and full-duplex capabilities. It targets engineers and researchers evaluating state-of-the-art multimodal AI, offering a transparent and concise demonstration of the model's potential benefits in real-time interactive applications.

How It Works

The system employs an end-to-end omnimodal architecture, densely connecting modality encoders/decoders with the LLM via hidden states for enhanced information flow. A key innovation is the Full-Duplex Omni-modal Live Streaming Mechanism, which transforms offline components into online, full-duplex processors capable of handling simultaneous streaming inputs and outputs. This is achieved through interleaved text/speech token generation and a time-division multiplexing (TDM) mechanism for synchronized omni-modality processing. Additionally, a Proactive Interaction Mechanism allows the LLM to continuously monitor streams and autonomously decide when to speak, enabling more dynamic and responsive AI behavior.

Quick Start & Requirements

The recommended deployment path is via Docker Compose. Key requirements include an NVIDIA GPU with over 28GB of VRAM, a Linux operating system, Docker with the Compose v2 plugin, and the NVIDIA Container Toolkit. Model weights must be mounted from the host. Ready-to-use desktop installers for Windows and macOS are available from the llama.cpp-omni Releases page. Documentation and community links (Discord, Feishu Group) are provided.

Highlighted Details

• Leading Visual Capability: Achieves an average score of 77.6 on OpenCompass, outperforming models like GPT-4o and Gemini 2.0 Pro in vision-language tasks, while supporting instruct and thinking modes.
• Advanced Speech: Offers bilingual (English/Chinese) real-time speech conversation with voice cloning and role-play features that surpass tools like CosyVoice2.
• Full-Duplex Streaming: Enables simultaneous processing of real-time video and audio streams with concurrent text and speech output, facilitating fluid, non-blocking omnimodal conversations and proactive interactions.
• High-Performance OCR: Processes high-resolution images (up to 1.8 million pixels) and high-FPS videos (up to 10fps), achieving state-of-the-art English document parsing on OmniDocBench and supporting over 30 languages.
• Flexible Deployment: Supports various inference backends including PyTorch, llama.cpp, vLLM, and SGLang, along with quantization formats like int4 and GGUF.

Maintenance & Community

Community channels include Discord and a Feishu Group. Specific details on core maintainers, sponsorships, or partnerships are not provided in the README.

Licensing & Compatibility

The specific open-source license for this project is not explicitly stated in the provided documentation, which is a critical omission for adoption decisions. Compatibility is high due to support for multiple inference engines and quantization formats.

Limitations & Caveats

The primary deployment backend (PyTorch) requires significant GPU resources (NVIDIA GPU with >28GB VRAM) and a Linux environment. The lack of explicit licensing information presents a major adoption blocker, requiring clarification before commercial or widespread use.