This repository provides the PARTNR benchmark for evaluating Large Planning Models (LPMs) in human-robot collaboration and instruction-following tasks within the Habitat simulator. It offers a framework for agents to interpret natural language commands and interact with simulated environments, targeting researchers and developers in embodied AI and robotics.

How It Works

The system models agents, tools, and skills within the Habitat simulator. Agents interact with the environment using a set of tools, which abstract perception and low-level actions. Planners, often LLM-based, leverage a hierarchical WorldGraph representing the environment's state to select appropriate tools for task execution. A simulated perception pipeline feeds local detections into the world model, enabling agents to reason about their surroundings.

Quick Start & Requirements

• Install/Run: Use python -m habitat_llm.examples.planner_demo with specified config files (e.g., baselines/decentralized_zero_shot_react_summary.yaml).
• Prerequisites: Python, Habitat simulator, LLM access (e.g., Hugging Face models like Llama-3.8B, or OpenAI API keys). Neural network skills require downloading checkpoints.
• Resources: Requires LLM inference capabilities. Specific hardware requirements depend on the chosen LLM.
• Links: Project Website, Paper Overview

Highlighted Details

• Supports decentralized and centralized multi-agent planning.
• Enables zero-shot and heuristic planning approaches.
• Allows custom instruction execution and OpenAI backend integration.
• Includes tools for dataset visualization, analysis, and episode verification.

Maintenance & Community

• Developed by Facebook AI Research (FAIR).
• Codebase includes unit tests and example scripts.
• Citation details for the associated paper are provided.

Licensing & Compatibility

• MIT License.
• Permissive for commercial use and integration with closed-source projects.

Limitations & Caveats

The skill_runner currently supports only sequential skill execution for one agent at a time, limiting simultaneous multi-agent interaction within that specific tool. Constrained generation is not supported for the OpenAI backend.