This repository provides code for structural probes, a method to test whether neural network hidden states encode discrete linguistic structures like syntax. It is targeted at NLP researchers and engineers seeking to analyze and understand the internal representations of language models. The primary benefit is enabling quantitative evaluation of syntactic information captured by these models.

How It Works

Structural probes train linear models to predict syntactic properties (e.g., parse depth, distance between words) from neural network embeddings. The approach uses a probe that maps hidden states to a low-dimensional space, allowing for efficient estimation of syntactic relationships. This method is advantageous for its interpretability and ability to isolate specific linguistic phenomena within complex model representations.

Quick Start & Requirements

• Install dependencies: conda install --file requirements.txt and pip install pytorch-pretrained-bert. PyTorch must be installed separately according to your system's GPU/CPU configuration.
• Download example data: bash ./download_example.sh
• Demo: printf "The chef that went to the stores was out of food" | python structural-probes/run_demo.py example/demo-bert.yaml
• Experiment: python structural-probes/run_experiment.py example/config/prd_en_ewt-ud-sample.yaml
• Requires Python 3, GPU recommended.

Highlighted Details

• Supports ELMo and BERT models, with options for different probe types (distance, depth).
• Includes reporting methods for Spearman correlation, unlabeled attachment score (UAS), and visualizations of predicted vs. gold parse structures.
• Provides scripts for converting data and generating embeddings for replication of paper results.

Maintenance & Community

The project is associated with the paper "A Structural Probe for Finding Syntax in Word Representations" by John Hewitt and Christopher D. Manning. Further community or maintenance details are not explicitly provided in the README.

Licensing & Compatibility

The repository's license is not explicitly stated in the README. Compatibility for commercial use or closed-source linking is not specified.

Limitations & Caveats

The README notes that support for plugging in arbitrary PyTorch models is not yet streamlined. Replicating results on datasets like PTB requires significant data preprocessing steps involving external tools like Stanford CoreNLP and AllenNLP. The project is primarily geared towards researchers familiar with NLP data formats and model evaluation.