The ANLI dataset provides a benchmark for Natural Language Understanding (NLU) by presenting adversarial examples designed to challenge NLI models. It targets researchers in linguistics, machine learning, cognitive science, and psychology, offering a robust evaluation suite and pre-trained models for NLI tasks.

How It Works

ANLI is an adversarial dataset, meaning examples are specifically crafted to fool existing NLI models. It consists of three rounds of increasing difficulty, with annotations for error analysis and verifier labels to ensure quality. The dataset is formatted in JSONL and includes premises, hypotheses, labels (entailment, neutral, contradiction), and reasons for the labels.

Quick Start & Requirements

• Install: Use Hugging Face Transformers library.
• Prerequisites: Python 3.7, PyTorch 1.7, Transformers 3.0.2+.
• Usage: Load pre-trained models from Hugging Face Hub (e.g., ynie/roberta-large-snli_mnli_fever_anli_R1_R2_R3-nli).
• Resources: Google Colab notebooks are available for quick experimentation.
• Links: Dataset v1.0, Error Analysis, Leaderboard

Highlighted Details

• Features three rounds of adversarial examples (R1, R2, R3) with increasing difficulty.
• Includes detailed annotations for error analysis and verifier labels for improved interpretability.
• Offers pre-trained models on a combined dataset (SNLI, MNLI, FEVER-NLI, ANLI) via Hugging Face Hub.
• Supports multiple model architectures including RoBERTa, ALBERT, BART, ELECTRA, and XLNet.

Maintenance & Community

The project is maintained by Facebook AI Research (FAIR). The README encourages contributions via pull requests for leaderboard additions and model submissions.

Licensing & Compatibility

ANLI is licensed under Creative Commons-Non Commercial 4.0. This license restricts commercial use and linking with proprietary codebases.

Limitations & Caveats

The dataset's non-commercial license may limit its adoption in commercial products. The provided code and pre-trained models are based on older versions of PyTorch and Transformers, potentially requiring updates for compatibility with current libraries.