ReMEmbR enables robots to build and reason over long-horizon spatio-temporal memories using Large Language Models (LLMs) and Vision-Language Models (VLMs). It allows robots to answer complex queries about their environment and past experiences, such as navigation instructions or temporal event analysis. The target audience includes robotics researchers and developers working with embodied AI and memory-augmented systems.

How It Works

ReMEmbR integrates LLMs and VLMs with a persistent memory database, specifically MilvusDB, to store and retrieve spatio-temporal information. Memory items consist of captions (from VLMs), timestamps, and pose data. This approach allows for efficient querying and reasoning over extensive historical data, enabling more sophisticated robot behaviors and question-answering capabilities.

Quick Start & Requirements

• Install: Clone VILA, run ./vila_setup.sh, install Ollama, activate conda environment, pip install -r requirements.txt, and launch MilvusDB via bash launch_milvus_container.sh start.
• Prerequisites: Docker, Conda, GCC 12.1.0 (if GLIBCXX error occurs).
• Resources: Requires downloading models and potentially large datasets.
• Docs: VILA, MilvusDB, Isaac ROS Documentation.

Highlighted Details

• LLM-agnostic design, supporting models like Command-R.
• Integrates with ROS2 for real-world robot data pipelines.
• Includes Gradio demos for offline and live (Nova Carter) examples.
• Provides a NaVQA dataset for evaluation.

Maintenance & Community

• Developed by NVIDIA-AI-IOT.
• Mentions dependencies like WhisperTRT, MilvusDB, VILA, ROS2, Gradio, and LangGraph.
• Links to Isaac ROS Documentation and Jetson AI Lab.

Licensing & Compatibility

• Code directly authored in the repository is detailed in LICENSE.md.
• Project depends on third-party software with their own licenses (WhisperTRT, MilvusDB, VILA, ROS2, Gradio, LangGraph). Users must review all terms.

Limitations & Caveats

The project notes a potential GLIBCXX version error requiring GCC update. It also highlights that dependencies may download models or data, necessitating review of those components' licenses.