Summary

JoyAI-VL-Interaction offers a fully open-source, 8B-scale vision-language interaction system designed to overcome the limitations of turn-based AI. It provides a proactive, real-time presence, continuously observing visual input and responding only when necessary, making it suitable for scenarios where immediate AI assistance is crucial. The project includes the model, training recipe, time-aligned data, and a deployable streaming stack, targeting researchers and engineers building next-generation interactive AI.

How It Works

The core of the system is an 8B-scale, vision-first model that makes a real-time decision every second: to speak, stay silent, or delegate tasks. It employs a predictive video codec (AdaCodec) to efficiently manage token budgets for long video streams by focusing computational resources on significant scene changes. The model's behavior is learned from over four million time-aligned interaction clips and further refined through reinforcement learning, enabling natural presence and proactivity.

Quick Start & Requirements

To get started, clone the repository, install dependencies using ./install/install.sh --with-all, download model weights with ./install/download-models.sh --all, and run the core services via ./services/scripts/run.sh minimal. The WebUI can then be accessed at https://127.0.0.1:8099. A full setup, including ASR, TTS, and a background agent, requires consulting the doc/getting_started.md guide.

Highlighted Details

• Real-time Presence: Continuously watches video streams and responds in under a second when critical moments arise.
• Vision-triggered Proactivity: Initiates communication based on visual cues, remaining silent when no action is needed.
• Agent Delegation: Capable of offloading complex subtasks to external models or agents while maintaining stream observation.
• Fully Open Stack: Provides open access to model weights, training recipe, interaction data, and a complete deployable system for reproducibility.
• Competitive Evaluation: Demonstrates strong performance in event-driven visual interaction settings, outperforming Doubao and Gemini in key aspects like monitoring, real-time counting, and long visual memory.

Maintenance & Community

As of its full open-source release on June 20, 2026, specific details on ongoing maintenance, notable contributors, or community channels (like Discord/Slack) are not yet detailed in the README.

Licensing & Compatibility

The project is licensed under the Apache License 2.0, which permits commercial use and integration into closed-source projects.

Limitations & Caveats

JoyAI-VL-Interaction is an 8B-scale model and does not aim to match the broad capabilities or polish of much larger, commercially developed models like Doubao or Gemini in areas such as open-ended chat or handling the long tail of everyday requests. Its strength lies in its specialized domain of real-time, proactive vision-language interaction.