ReKep addresses the challenge of generating closed-loop robotic manipulation trajectories by leveraging large vision and vision-language models within a hierarchical optimization framework. It is designed for researchers and engineers working on robotic manipulation, offering a method to create robust, adaptive motion plans for complex tasks.

How It Works

ReKep employs a hierarchical optimization approach, using vision-language models (VLMs) to generate relational keypoint constraints. These constraints are then fed into an optimization framework to derive closed-loop trajectories. This method allows for spatio-temporal reasoning, enabling the robot to adapt to dynamic environments and recover from disturbances by replanning based on real-time keypoint information.

Quick Start & Requirements

• Install OmniGibson (tested commit specified).
• Requires Isaac Sim installed in a default directory.
• Set OPENAI_API_KEY environment variable.
• Primary run command: python main.py --use_cached_query [--visualize]
• Recommended for use with a display; headless mode instructions are available.
• Demo available for "pen-in-holder" task.
• Links: [Project Page] [Paper] [Video]

Highlighted Details

• Hierarchical optimization framework using VLMs for constraint generation.
• Closed-loop trajectory generation for robotic manipulation.
• Robustness to disturbances with automatic failure recovery.
• Supports task configuration with OmniGibson's BEHAVIOR-1K assets.

Maintenance & Community

• Project associated with Stanford University and Columbia University.
• OmniGibson issues can be raised on their repository.
• Discord channel available for OmniGibson support.

Licensing & Compatibility

• No explicit license mentioned in the README.
• Codebase is implemented in OmniGibson, which has its own licensing.

Limitations & Caveats

The provided codebase does not include the perception pipeline (keypoint tracking, mask tracking, SDF reconstruction) used in real-world experiments; it relies on simulation data. Real-world deployment requires significant implementation of robot controllers, keypoint trackers, and SDF reconstruction. The method is sensitive to VLM choice and prompt engineering. Performance tuning may be necessary due to sequential pipeline execution and solver latency. Planning is done in task space, which can occasionally lead to kinematically challenging motions.