This project provides a conversational application for the Reachy Mini robot, enabling users to interact with it through speech and vision. It targets developers and researchers looking to build sophisticated, AI-driven robotic behaviors by integrating real-time voice processing, vision pipelines, and choreographed motion. The application offers a flexible and extensible framework for creating dynamic and responsive robot interactions.

How It Works

The app employs a layered architecture for real-time interaction. A core component is the real-time audio conversation loop utilizing fastRTC for low-latency streaming, supporting Hugging Face, OpenAI, and Gemini LLM backends. Vision processing integrates with these backends or can leverage on-device SmolVLM2 models. A layered motion system manages primary robot movements while blending speech-reactive elements and head-tracking. Interaction is facilitated through an Async tool dispatch system, accessible via a Gradio web UI, which orchestrates robot motion, camera capture, and other capabilities.

Quick Start & Requirements

• Primary install / run command: Recommended installation uses uv (uv venv --python python3.12 .venv, source .venv/bin/activate, uv sync). Alternatively, use pip (python -m venv .venv, source .venv/bin/activate, pip install -e .). The application is launched via reachy-mini-conversation-app.
• Non-default prerequisites and dependencies: Requires Reachy Mini's SDK to be installed. Python 3.12 is recommended. Optional features like local_vision, yolo_vision, mediapipe_vision, and remote_tools require specific installation extras (e.g., uv sync --extra local_vision or pip install -e .[local_vision]). GPU is recommended for local_vision.
• Estimated setup time or resource footprint: Suggested storage is 'large'. Setup involves standard Python environment configuration.
• Links: No direct external links to quick-start guides or demos are provided; the repository's README serves as the primary documentation.

Highlighted Details

• Supports multiple LLM backends: Hugging Face (default), OpenAI, and Gemini, configurable via environment variables.
• Offers flexible vision processing options, including real-time backends and on-device SmolVLM2 (CPU/GPU/MPS) via the --local-vision flag.
• Features an extensible system for custom profiles, allowing modification of instructions, enabled tools, and custom Python tool implementations.
• Enables integration of public Hugging Face Space tools as remote action sources using the tool-spaces command.

Maintenance & Community

The repository includes a contribution guide. No specific community channels (e.g., Discord, Slack) or notable contributors/sponsorships are mentioned in the provided text.

Licensing & Compatibility

• License type: Apache 2.0.
• Compatibility notes: The Apache 2.0 license is permissive and generally compatible with commercial use and closed-source linking.

Limitations & Caveats

Windows support is noted as experimental and requires cautious use. The --local-vision feature is not supported when running the conversation app directly on Reachy Mini Wireless or Raspberry Pi; it necessitates running the app from a separate workstation.