OpenXAI is a lightweight, general-purpose library designed to systematically evaluate the quality of attribute-based model explanations. It targets researchers and practitioners in Explainable AI (XAI), offering a comprehensive suite of tools to promote transparent, reproducible, and systematic evaluation of explanation methods. The library benefits users by providing a unified framework for benchmarking, accelerating research, and fostering transparency through public leaderboards.

How It Works

OpenXAI operates as an open-source ecosystem encompassing XAI-ready datasets (both synthetic and real-world), implementations of state-of-the-art explanation methods, a collection of evaluation metrics, and public leaderboards. It provides straightforward API interfaces for loading datasets, pre-trained models, generating explanations, and benchmarking them against various metrics. This unified approach facilitates the systematic and efficient evaluation of existing and new explanation methods, thereby informing and accelerating research in the emerging field of XAI.

Quick Start & Requirements

• Primary install command: pip install -e . (after cloning the repository).
• Prerequisites: Core environment dependencies are installed via pip. No specific hardware (e.g., GPU, CUDA) or Python version is explicitly mandated for basic installation.
• Relevant links: Website, arXiv Paper.

Highlighted Details

• Integrates 7 state-of-the-art feature attribution methods and 22 quantitative evaluation metrics.
• Features a flexible synthetic data generator for creating diverse datasets and facilitating the construction of ground truth explanations.
• Hosts the first public XAI leaderboards to promote transparency and enable direct comparison of explanation method performance.
• Supports easy integration of custom explanation methods and evaluation metrics via class extension or form submission.

Maintenance & Community

• Contact: openxaibench@gmail.com or by opening a GitHub issue.
• The project is associated with authors from the NeurIPS 2022 Datasets and Benchmarks Track paper.

Licensing & Compatibility

• License type: MIT license for the codebase.
• Compatibility: The MIT license is generally permissive for commercial use and closed-source linking. Usage of individual datasets requires adherence to their respective licenses.

Limitations & Caveats

The project is at version 0.0.0, indicating an early stage of development. While comprehensive, the focus is on attribute-based explanation methods, and users may need to refer to individual dataset licenses for usage terms.