Language model probe for factual/commonsense knowledge analysis (research paper)
Top 29.8% on sourcepulse
LAMA is a probe for analyzing factual and commonsense knowledge within pretrained language models. It offers a unified interface to query models like BERT, RoBERTa, ELMo, and Transformer-XL, enabling researchers and practitioners to assess model capabilities and extract knowledge.
How It Works
LAMA operates by presenting language models with cloze-style prompts (e.g., "The capital of France is [MASK].") and analyzing their predictions. It leverages a dataset of such prompts designed to test specific factual and commonsense knowledge. The project provides connectors to various popular language model architectures, abstracting away model-specific APIs for consistent analysis.
Quick Start & Requirements
pip install -r requirements.txt
(after cloning and setting up a conda environment with Python 3.7).download_models.sh
), spaCy (python3 -m spacy download en
), and the LAMA dataset (data.zip
).Highlighted Details
pip install -e git+https://github.com/facebookresearch/LAMA#egg=LAMA
).Maintenance & Community
Licensing & Compatibility
Limitations & Caveats
1 year ago
1 day