This repository provides a unified and extensible framework for evaluating Large Language Model (LLM) unlearning. It supports popular benchmarks like TOFU and MUSE, offering implementations for multiple unlearning methods, datasets, evaluation metrics, and LLM architectures, making it a valuable resource for researchers and practitioners in LLM privacy and security.

How It Works

Open-Unlearning leverages Hydra for flexible experiment configuration, allowing users to easily select and combine benchmarks, unlearning methods, datasets, and model architectures. It provides streamlined implementations of unlearning algorithms and evaluation metrics, facilitating reproducible research and benchmarking. The framework is designed for extensibility, encouraging community contributions of new components.

Quick Start & Requirements

• Install:

  conda create -n unlearning python=3.11
  conda activate unlearning
  pip install .
  pip install --no-build-isolation flash-attn==2.6.3
  

• Data Setup: Run python setup_data.py to download evaluation log files.
• Prerequisites: Python 3.11, Conda, FlashAttention v2.6.3.
• Documentation: Overview, Quickstart, Experiments, Evaluation.

Highlighted Details

• Supports TOFU and MUSE benchmarks with 6 unlearning methods (GradAscent, GradDiff, NPO, SimNPO, DPO, RMU).
• Includes 9+ evaluation metrics, including 6 Membership Inference Attacks (MIA), Extraction Strength (ES), and Exact Memorization (EM).
• Offers fine-tuned models for TOFU benchmark across LLaMA 3.2, LLaMA 3.1, and LLaMA-2 architectures.
• Features an extensible design for adding new benchmarks, methods, datasets, and metrics, with a community leaderboard.

Maintenance & Community

• Developed by Vineeth Dorna and Anmol Mekala.
• Encourages community contributions via contributing.md.
• Leaderboard for tracking results.

Licensing & Compatibility

• MIT License.
• Compatible with commercial use and closed-source linking.

Limitations & Caveats

The repository is a replacement for the no-longer-maintained locuslab/tofu repository. Users must run setup_data.py after merging the latest version to refresh evaluation log files.