AesBench: An Expert Benchmark for MLLM Aesthetic Perception

AesBench addresses the underexplored capacity of Multimodal Large Language Models (MLLMs) to understand image aesthetics, a gap hindering applications in art and design. It targets researchers and developers evaluating MLLMs, providing a systematic benchmark to assess aesthetic perception, empathy, assessment, and interpretation. This enables deeper investigation into MLLMs' nuanced visual understanding.

How It Works

The benchmark constructs a dataset using high-quality annotations from aesthetic experts. It employs integrative criteria to evaluate MLLMs across four perspectives: perception (AesP), empathy (AesE), assessment (AesA), and interpretation (AesI). This multi-faceted approach allows for a comprehensive assessment of aesthetic comprehension beyond superficial image analysis.

Quick Start & Requirements

The evaluation database and codes are released, with the dataset available on Huggingface. AesBench is integrated into the VLMEvalKit for convenient testing. Specific installation commands and detailed prerequisites are not explicitly detailed in the README, but MLLM evaluation tooling is implied.

• Dataset: https://huggingface.co/datasets/qyuan/EAPD_release
• Evaluation Toolkit: https://github.com/open-compass/VLMEvalKit
• Leaderboard: https://aesbench.github.io/

Highlighted Details

• Supports a broad spectrum of closed-source models including GPT-4v, GPT-4o, Gemini-1.0-Pro, and Claude3-Opus, alongside numerous open-source MLLMs like LLaVA, InternVL2, and Qwen-VL.
• Features a continuously updated leaderboard showcasing model performance on aesthetic evaluation.
• Offers integration with the VLMEvalKit for streamlined testing workflows.

Maintenance & Community

The project appears actively maintained, with recent updates in July 2024. It acknowledges contributions from aesthetic experts and development team members. No specific community channels (e.g., Discord, Slack) or direct contact links beyond the GitHub repository are provided.

Licensing & Compatibility

The project is licensed under the Apache-2.0 License. This permissive license allows for broad use, modification, and distribution, including in commercial applications, with minimal restrictions.

Limitations & Caveats

The project is actively developing, with a "TO DO" list indicating ongoing model integration efforts.