memit  by kmeng01

Transformer memory mass-editor (ICLR 2023 research paper)

created 2 years ago
509 stars

Top 62.1% on sourcepulse

GitHubView on GitHub
1 Expert Loves This Project
Project Summary

MEMIT enables efficient, large-scale factual editing within pre-trained transformer language models. It targets researchers and practitioners seeking to correct or update knowledge embedded in LLMs without full retraining. The primary benefit is the ability to modify thousands of facts with minimal computational overhead compared to fine-tuning.

How It Works

MEMIT operates by identifying and modifying specific weights within the transformer's attention layers. It formulates factual edits as targeted interventions, calculating necessary weight adjustments to steer the model's output towards the desired new fact. This approach avoids catastrophic forgetting and allows for precise, localized modifications.

Quick Start & Requirements

  • Install via bash ./scripts/setup_conda.sh $CONDA_HOME.
  • Requires Conda for environment management, PyTorch, and CUDA.
  • Demo notebook available at notebooks/memit.ipynb.
  • Evaluation suite at experiments/evaluate.py.

Highlighted Details

  • Capable of editing thousands of facts in a single pass.
  • Demonstrates effectiveness on models like GPT-J-6B.
  • Provides tools for full evaluation and result summarization.

Maintenance & Community

The project is associated with ICLR 2023 and authored by Kevin Meng et al. No specific community channels or active maintenance signals are provided in the README.

Licensing & Compatibility

The project is released under an unspecified license. The README does not detail licensing terms or compatibility for commercial use.

Limitations & Caveats

The README does not specify the license, making commercial use uncertain. It also lacks explicit details on supported model architectures beyond the GPT-J-6B example.

Health Check
Last commit

1 year ago

Responsiveness

1+ week

Pull Requests (30d)
0
Issues (30d)
0
Star History
25 stars in the last 90 days

Explore Similar Projects

Feedback? Help us improve.