Captum is a PyTorch library for model interpretability, offering a suite of algorithms to understand feature importance, neuron contributions, and model behavior. It targets ML researchers and developers seeking to debug, improve, and explain their models, providing insights into what drives predictions.

How It Works

Captum implements various attribution algorithms, including Integrated Gradients, DeepLift, and GradientShap, which attribute model predictions to input features or internal model components. It leverages PyTorch's autograd capabilities to efficiently compute these attributions, often using baseline inputs or noise injection for robustness and accuracy. The library supports extensions for layer and neuron-level analysis, enabling deeper inspection of model internals.

Quick Start & Requirements

• Install via pip: pip install captum
• Install via conda: conda install captum -c pytorch or conda install captum -c conda-forge
• Requirements: Python >= 3.9, PyTorch >= 1.10.
• For development: pip install -e .[dev] or pip install -e .[tutorials]
• Official documentation and tutorials are available.

Highlighted Details

• Implements state-of-the-art interpretability algorithms like Integrated Gradients, TCAV, and TracIn.
• Offers noise tunneling (SmoothGrad, VarGrad) for attribution smoothing.
• Provides neuron and layer conductance for internal model analysis.
• Includes adversarial attack and minimal input perturbation capabilities.

Maintenance & Community

• Developed by the PyTorch team.
• Active development and contributions from the community.
• Resources include an FAQ and contribution guidelines.

Licensing & Compatibility

• BSD 3-Clause License.
• Permissive license suitable for commercial use and integration into closed-source projects.

Limitations & Caveats

• Captum Insights (visualization tool) is being deprecated.
• Some algorithms, like DeepLift, have limitations on supported activation types and may require careful baseline selection.