The Numenta Anomaly Benchmark (NAB) provides a standardized framework and dataset for evaluating anomaly detection algorithms in real-time, streaming applications. It targets researchers and developers building anomaly detection systems, offering a comprehensive corpus of labeled time-series data and a novel scoring mechanism to compare algorithm performance.

How It Works

NAB utilizes a corpus of 58 labeled real-world and artificial time-series datasets, covering diverse sources like server metrics and social media trends. It employs a unique scoring mechanism designed for real-time applications, which normalizes performance based on detection accuracy and penalizes false positives and false negatives. This approach allows for objective comparison of different anomaly detection algorithms.

Quick Start & Requirements

• Install: pip install -r requirements.txt followed by pip install . --user
• Prerequisites: Python 3.6, pip, NumPy.
• Supported Platforms: OSX 10.9+, Amazon Linux. Windows 10 is not officially supported.
• Additional Info: NAB Wiki

Highlighted Details

• Comprehensive corpus of 58 labeled real-world and artificial time-series data files.
• Novel scoring mechanism for real-time anomaly detection, with detailed performance comparisons against various algorithms.
• Supports integration of custom algorithms via defined entry points.
• Includes scripts for plotting results and running specific detectors or subsets of data.

Maintenance & Community

• NAB v1.1 is the latest version.
• Contact: nab@numenta.org for contributions or inquiries.
• Scoreboard: https://github.com/numenta/NAB/wiki/Scoreboard

Licensing & Compatibility

• License: Not explicitly stated in the README, but the project is hosted on GitHub under a permissive structure. The NuPIC dependency is available on PyPI.

Limitations & Caveats

• Some benchmarked detectors (e.g., Numenta HTM, HTM Java, Twitter ADVec) are not Python 3 compatible and require separate execution environments or Docker images.
• Running the full NAB suite, including slower detectors like Skyline, can take many hours.
• Debugging with subsets of data may produce warnings regarding missing labels.