Octo provides a transformer-based generalist robotic policy (GRP) trained on 800k robot trajectories, enabling zero-shot control via language or goal images across diverse robot setups. It's designed for researchers and practitioners in robotics and AI who need a versatile, adaptable policy for various manipulation tasks.

How It Works

Octo employs a modular attention structure within its transformer backbone. This design allows it to process multimodal inputs (RGB cameras, language, goal images) and output robot actions. The modularity facilitates efficient finetuning on new robot morphologies, sensory inputs, and action spaces with minimal data and compute.

Quick Start & Requirements

• Install via pip install -e . and pip install -r requirements.txt.
• GPU users require jax[cuda11_pip] (version 0.4.20 specified).
• TPU users require jax[tpu] (version 0.4.20 specified).
• See installation details: Jax Github.
• Test installation with python scripts/finetune.py --config.pretrained_path=hf://rail-berkeley/octo-small-1.5 --debug.

Highlighted Details

• Supports multiple RGB camera inputs, various robot arms, and language/goal image instructions.
• Achieves 13 it/sec inference on a 1x NVIDIA 4090 for the 93M parameter Octo-Base model.
• Offers example scripts for inference, finetuning, rollouts, and real-robot evaluation.
• Pretraining requires ~1.2TB of data and significant compute (TPUv4-128 pod for 8-14 hours).

Maintenance & Community

• Developed by the Octo Model Team, including researchers from UC Berkeley.
• Citation available for academic use.

Licensing & Compatibility

• The repository does not explicitly state a license in the README.

Limitations & Caveats

• Pretraining requires substantial data (1.2TB) and compute resources.
• The README does not specify a license, which may impact commercial use or closed-source integration.