Quantus is an open-source toolkit for the quantitative evaluation of neural network explanations (XAI methods). It provides over 35 metrics across six categories—faithfulness, robustness, localization, complexity, randomization, and axiomatic—to help researchers and practitioners assess the quality and reliability of XAI outputs. The library supports PyTorch and TensorFlow models and various data types, aiming to facilitate responsible AI development.

How It Works

Quantus offers a standardized framework for applying and comparing XAI evaluation metrics. Users can either provide pre-computed explanations or pass an explanation function to the library. The core design allows for flexible metric instantiation with customizable parameters and supports batch processing for efficiency. This approach enables holistic quantification and sensitivity analysis of XAI methods, moving beyond qualitative visual comparisons.

Quick Start & Requirements

• Install via pip: pip install quantus or with framework support: pip install "quantus[torch]" or pip install "quantus[tensorflow]".
• Requires Python >= 3.8, PyTorch >= 1.11.0, and TensorFlow >= 2.5.0.
• Tutorials and examples are available via Google Colab and MyBinder.
• Official documentation: quantus.readthedocs.io

Highlighted Details

• Supports over 35 metrics across 6 categories for comprehensive XAI evaluation.
• Integrates with popular XAI libraries like Captum, tf-explain, and zennit.
• Offers a batch implementation for a 12x speedup on faithfulness metrics.
• Includes new metrics like EfficientMPRT and SmoothMPRT.

Maintenance & Community

The project is actively developed, with recent updates including new metrics and performance improvements. Contributions are welcomed, and a Discord server is available for community interaction. Contact: hedstroem.anna@gmail.com.

Licensing & Compatibility

Released under the MIT License, allowing for commercial use and integration with closed-source projects.

Limitations & Caveats

The library is under active development, and metric implementations have not been verified by original authors. Users are advised to note release versions for reproducibility and consult user guidelines for best practices.