WeightWatcher is a diagnostic tool for analyzing Deep Neural Networks (DNNs) without requiring access to training or test data. It helps users predict model accuracy, identify over-training or over-parameterization, and detect potential issues during compression or fine-tuning, targeting researchers and practitioners in deep learning.

How It Works

The tool is based on theoretical research into Heavy-Tailed Self-Regularization (HT-SR) and employs concepts from Random Matrix Theory (RMT) and Statistical Mechanics. It analyzes the Empirical Spectral Density (ESD) of layer weight matrices, fitting the tail of the distribution to a power law to derive generalization metrics. This approach aims to quantify how "on-random" or "heavy-tailed" a layer's weight distribution is, correlating these properties with model generalization performance.

Quick Start & Requirements

• Install via pip: pip install weightwatcher
• Requires Python 3.7+, PyTorch 1.x, and TensorFlow 2.x/Keras.
• Supported layers include Dense/Linear, Fully Connected, Conv1D, and Conv2D.
• Official blog and example notebooks are available for deeper dives.

Highlighted Details

• Predicts test accuracies and identifies over/under-training without data.
• Analyzes PEFT/LoRA models by examining delta layers.
• Detects "Correlation Traps" indicative of poor training via randomized ESD comparison.
• Offers metrics like alpha (power law exponent) and rand_distance for generalization assessment.

Maintenance & Community

• Developed by Charles H. Martin (Calculation Consulting) and contributors.
• Active Discord server for community support and discussion.
• Numerous academic papers and presentations detail the underlying research.

Licensing & Compatibility

• Licensed under the Apache License 2.0.
• Compatible with commercial use and closed-source linking.

Limitations & Caveats

The power law fits are most effective for well-trained, heavy-tailed ESDs; results may be spurious if alpha > 8.0 or if ESDs are multimodal or not well-described by a single power law. The PEFT/LoRA analysis is experimental and currently ignores biases.