This project provides a community-driven Jupyter Notebook to track progress in AI and machine learning research by collecting problems, metrics, and datasets from the literature. It aims to inform journalists, policymakers, and users about the state of AI, enabling proactive risk assessment and policy development.

How It Works

The project utilizes a Jupyter Notebook to aggregate and present data points, where each .measure() call represents an algorithm's performance on a specific metric/dataset. This approach allows for a granular view of progress across various AI subfields, drawing inspiration and data from existing collections of AI progress measurements.

Quick Start & Requirements

• Install: Clone the repository and install dependencies via pip install cssselect lxml matplotlib{,-venn} numpy requests seaborn.
• Prerequisites: Git, Jupyter Notebook (version 3+).
• Contribution: Configure git with ipynb_drop_output for merge filters. Run jupyter notebook in the project directory to edit AI-progress-metrics.ipynb.
• Alternative: Use Microsoft Azure's Notebook service (with caveats regarding arXiv access and Unicode handling).
• Docs: https://eff.org/ai/metrics

Highlighted Details

• Community-driven effort to document AI progress through data.
• Aggregates data from various sources, including academic papers and existing AI progress collections.
• Data is available in JSON format (progress.json) for external use and visualization.
• Focuses on frontier performance but also includes notable algorithms and field progress.

Maintenance & Community

Initial authors are from EFF, with contributions from various individuals and inspiration from organizations like OpenAI and the Future of Humanity Institute. The project aims to grow into a self-sustaining community effort.

Licensing & Compatibility

The repository is open source, with the specific license not explicitly stated in the provided text, but the nature of the project suggests a permissive license suitable for community contribution and data sharing.

Limitations & Caveats

The project is primarily a data collection effort and does not track training speed or efficiency. Some data points may require double-checking for accuracy regarding paper revisions. The Azure Notebooks service has limitations with arXiv requests and Unicode handling.