This repository provides a PyTorch implementation for "Driving with LLMs," a system that fuses object-level vector data with pre-trained Large Language Models (LLMs) to predict explainable autonomous driving actions. It targets researchers and engineers in autonomous driving, offering a robust and interpretable approach to decision-making.

How It Works

The LLM-Driver utilizes object-level vector inputs from a driving simulator, feeding them into pre-trained LLMs. This approach allows for the prediction of steering angles and acceleration/braking commands, alongside generating natural language justifications for these actions and answering driving-related questions. This fusion of structured vector data and LLM capabilities aims to enhance the explainability and interpretability of autonomous driving systems.

Quick Start & Requirements

• Install: pip install -r requirements.txt.lock
• Prerequisites: Python 3.x, minimum 20GB VRAM for evaluation, 40GB VRAM for training. WandB API key required for logging.
• Dataset: Unarchive data/vqa_train_10k.tar.gz and data/vqa_test_1k.tar.gz.
• Links: Paper, LingoQA

Highlighted Details

• Implements the LLM-Driver for explainable autonomous driving.
• Supports open-loop prediction, action prediction, and Driving Question Answering (VQA).
• Includes scripts for data collection using OpenAI ChatGPT API and grading VQA results.
• Offers training and evaluation for both the LLM-Driver and a Perceiver-BC model.

Maintenance & Community

The project is associated with authors from ICRA 2024 and has a follow-up work, LingoQA. It draws inspiration from the Alpaca LoRA repository.

Licensing & Compatibility

The repository does not explicitly state a license in the provided README.

Limitations & Caveats

The project requires significant VRAM (20GB for evaluation, 40GB for training), which may be a barrier for users with limited hardware. The absence of an explicit license could pose compatibility issues for commercial use or closed-source integration.