OpenEMMA is an open-source framework for end-to-end motion planning in autonomous driving, replicating Waymo's EMMA model. It targets researchers and developers, leveraging pre-trained Vision Language Models (VLMs) to integrate text and camera inputs for predicting future waypoints and generating decision rationales.

How It Works

OpenEMMA integrates multimodal inputs, including front-view camera images and textual descriptions, with large Vision Language Models (VLMs) like GPT-4 and LLaVA. This approach allows the model to understand complex driving scenarios, predict future trajectories, and provide human-readable explanations for its decisions, aiming for more interpretable and robust autonomous driving systems.

Quick Start & Requirements

• Install via PyPI: pip install openemma
• Alternatively, clone the repo and install dependencies: pip install -r requirements.txt
• Requires Python 3.8 and cudatoolkit (tested with CUDA 12.4.0).
• Requires OpenAI API key for GPT-4 integration.
• Requires the nuScenes dataset.
• See: Installation Guide

Highlighted Details

• Supports multiple VLMs: GPT-4o, LLaVA-1.6-Mistral-7B, Llama-3.2-11B-Vision-Instruct, Qwen2-VL-7B-Instruct.
• Generates predicted waypoints, decision rationales, annotated images, and compiled videos.
• Integrates YOLO-3D for critical object detection.

Maintenance & Community

• Project initiated by taco-group.
• For help, submit a GitHub issue. Contact: Shuo Xing (shuoxing@tamu.edu).
• Paper available: arXiv:2412.15208

Licensing & Compatibility

• Code License: Apache 2.0.
• Permissively licensed, suitable for commercial use and integration into closed-source projects.

Limitations & Caveats

The project is presented as a reproduction and may require significant computational resources for training and inference, particularly for larger VLM models. The README does not specify performance benchmarks or hardware requirements beyond CUDA.