This repository provides the codebase for EmbodiedGPT, a vision-language pre-training model that leverages embodied chain-of-thought reasoning. It is designed for researchers and engineers working on multimodal AI, offering a flexible framework for training on diverse datasets including images, videos, and text.

How It Works

The core of the library is built around PyTorch's Dataset and DataLoader. It introduces BaseDataset for handling heterogeneous media types (images, videos, text) with standardized transformations and task-specific processing. The WeightedConcatDataset allows for combining multiple datasets with adjustable weights, enabling balanced training across different data sources and tasks. This modular design facilitates customization and integration into existing PyTorch training pipelines.

Quick Start & Requirements

• Installation: Follow instructions in INSTALLATION.md.
• Data Preparation: Download EgoCOT and COCO-2017 datasets. Unzip datasets_share.zip to ./datasets/.
• Pretrained Model: Download Embodied_family_7btiny.
• Prerequisites: PyTorch, Python. Specific versions and hardware requirements are detailed in INSTALLATION.md.

Highlighted Details

• Enables training on heterogeneous data (images, videos, text) using PyTorch's Dataset and DataLoader.
• WeightedConcatDataset allows for weighted combination of multiple datasets.
• Designed for flexibility and customization through subclassing BaseDataset.

Maintenance & Community

The project is associated with the paper "EmbodiedGPT: Vision-language pre-training via embodied chain of thought" published in NeurIPS 2024.

Licensing & Compatibility

• License: Apache 2.0.
• Compatibility: Permissive license suitable for commercial use and integration with closed-source projects.

Limitations & Caveats

The README indicates that instructions will be updated soon, suggesting the documentation might be incomplete or subject to change.