DriveDreamer is a pioneering world model for autonomous driving, built entirely from real-world driving scenarios. It addresses the limitations of existing models that focus on simulations by leveraging diffusion models to represent complex driving environments and a two-stage training pipeline to understand traffic constraints and predict future states. This enables precise, controllable driving video generation and the creation of realistic driving policies, targeting researchers and developers in autonomous driving.

How It Works

DriveDreamer utilizes a diffusion model to construct a comprehensive representation of complex driving environments, overcoming the overwhelming search space of intricate scenes. Its two-stage training pipeline first enables the model to acquire a deep understanding of structured traffic constraints, followed by a stage that equips it with the ability to anticipate future states. This approach allows for faithful capture of real-world traffic structures and generation of realistic driving policies.

Quick Start & Requirements

• Installation: Follow the provided installation guide.
• Prerequisites: Requires preparation of datasets and environment setup. Specific dependencies are detailed in the installation guide.
• Resources: The project is demonstrated on the nuScenes benchmark.
• Links:
  • Project Page: https://github.com/JeffWang987/DriveDreamer
  • Demos: https://github.com/JeffWang987/DriveDreamer/assets/49095445/a1f658ff-3ddc-4ec8-9e1f-9d3fe7183350

Highlighted Details

• First world model established from real-world driving scenarios.
• Achieves precise, controllable video generation capturing real-world traffic constraints.
• Enables generation of realistic and reasonable driving policies.
• Ranked 15th in Most Influential ECCV'24 Papers.

Maintenance & Community

• The research code was released on November 2, 2024.
• Future updates are available by following GigaAI-research.
• Related projects include DriveDreamer4D and DriveDreamer-2.

Licensing & Compatibility

• The project is associated with an ECCV 2024 paper, suggesting academic use. The specific license is not explicitly stated in the README.

Limitations & Caveats

The README does not explicitly detail limitations, unsupported platforms, or known bugs. The project is presented as research code, implying potential for ongoing development and changes.