RAI is a vendor-agnostic agentic framework for robotics, designed to integrate Generative AI features into existing ROS 2 robot stacks. It targets robotics developers and researchers seeking to enhance robots with natural language interaction, multi-modal understanding, and complex task execution capabilities. The framework aims to provide flexibility, human interactivity, and out-of-the-box AI functionalities.

How It Works

RAI leverages a multi-agent system architecture, integrating with tools like LangChain to abstract AI model vendors and provide access to various AI capabilities. It supports multi-modal inputs (including vision via VLMs) and can interact with ROS 2 interfaces, such as calling actions and publishing messages. The framework allows for customizable robot identities, including ethical constitutions and self-documentation, and can summarize robot states from ROS logs.

Quick Start & Requirements

• Installation: Requires ROS 2 (Jazzy or Humble) and Python 3.10 or 3.12. Installation is managed via Poetry.
  • Clone repository: git clone https://github.com/RobotecAI/rai.git && cd rai
  • Install dependencies: poetry install
  • Install ROS dependencies: rosdep install --from-paths src --ignore-src -r -y
  • Build: colcon build --symlink-install
  • Activate environment: source ./setup_shell.sh
• Configuration: Run poetry run streamlit run src/rai_core/rai/frontend/configurator.py to set up AI vendors and other settings.
• Optional Dependencies: Install with poetry install --with openset,nomad,s2s,simbench for specific features.
• Documentation: Setup instructions, Vendors Overview, Developer Guide.

Highlighted Details

• Supports voice interaction (both ways) and multi-modal capabilities (e.g., VLM integration).
• Enables natural language commands for ROS 2 actions and navigation goals (e.g., Nav2).
• Integrates with tools like Grounding SAM 2, GroundingDino, and NoMaD.
• Includes a debugging assistant for ROS 2 network state inspection.

Maintenance & Community

RAI is a focus project of the Embodied AI Community Group, with a talk presented at ROSCon 2024. A Discord server is available for community interaction.

Licensing & Compatibility

Licensed under Apache 2.0, permitting commercial use and integration with closed-source projects.

Limitations & Caveats

The framework acknowledges limitations inherent to LLMs/VLMs, including potential issues with spatial reasoning, hallucinations, latency, and costs. Resource usage (memory, CPU) is not yet optimized. Connectivity or an edge platform is required. The project is undergoing significant development for version 2.0, which will not be backward compatible with version 1.0.