This repository provides an implementation for evaluating text-to-3D generative models using GPT-4V as a human-aligned evaluator. It offers a framework for generating ELO scores for new methods by comparing them against existing ones, targeting researchers and developers in the 3D generation field.

How It Works

The core of the system leverages GPT-4V's ability to analyze rendered images of 3D models. For each text prompt, users must provide 120 RGB and normal map renderings of their generated 3D models. These renders are then fed to GPT-4V, which acts as an impartial judge, assigning scores that contribute to an ELO rating for the evaluated method. This approach aims to automate and standardize the qualitative evaluation process, which is often subjective and time-consuming.

Quick Start & Requirements

• Installation:
  • pip install --upgrade openai tqdm numpy Pillow gdown
  • PyTorch installation depends on your environment; refer to the official PyTorch website.
• Prerequisites: OpenAI API key, PyTorch, Python.
• Data: Download the tournament dataset (13 methods, 110 prompts, 120 renderings per prompt) from the provided Google Drive link.
• Evaluation: Organize your method's renders according to the specified folder structure and run python gpt_eval_alpha.py with appropriate arguments.
• Links: Dataset Download

Highlighted Details

• Utilizes GPT-4V for human-aligned evaluation of text-to-3D models.
• Generates ELO scores for comparative ranking of methods.
• Requires 120 RGB and normal map renderings per prompt at 512x512 resolution.
• Supports evaluation of new methods against a benchmark tournament.

Maintenance & Community

The project is associated with CVPR 2024 and acknowledges contributions from several prominent 3D generation projects, including threestudio, mvdream, and shap-e. Further utilities and visualization tools are planned.

Licensing & Compatibility

The repository's license is not explicitly stated in the README. However, given its reliance on OpenAI's API, users must adhere to OpenAI's terms of service. Compatibility with commercial or closed-source projects is not specified.

Limitations & Caveats

The evaluation process is dependent on the OpenAI API, which may incur costs and is subject to OpenAI's rate limits and availability. The quality of evaluation is directly tied to GPT-4V's capabilities and potential biases. The project is described as "alpha" in the evaluation command, suggesting potential instability or ongoing development.