BABILong: Long-Context LLM Evaluation Benchmark

BABILong addresses the critical challenge of evaluating Large Language Models (LLMs) in their ability to process and reason over extremely long documents. It targets researchers and engineers needing to assess LLM performance beyond typical context windows, providing a standardized benchmark to measure how effectively models can retrieve and utilize information embedded within vast amounts of irrelevant text. The primary benefit is a clearer understanding of LLM capabilities in real-world scenarios involving extensive documentation.

How It Works

BABILong employs a "needle-in-a-haystack" approach by embedding tasks from the bAbI dataset within large volumes of text sourced from the PG19 corpus. This creates documents potentially millions of tokens long, requiring models to distinguish crucial facts from extensive background noise. The benchmark comprises 20 distinct tasks designed to test various reasoning aspects, such as fact chaining, induction, deduction, and list/set handling, simulating scenarios where critical information is sparsely distributed.

Quick Start & Requirements

Evaluation examples are available in the ./notebooks and ./scripts directories, with data generation instructions in ./data. Pre-computed evaluation sets (100 and 1000 samples per task/length) and a leaderboard are hosted on Hugging Face Datasets. Specific hardware requirements (e.g., GPU, CUDA) are not explicitly detailed but are implied for running LLM evaluations.

Highlighted Details

• Features evaluation sets ranging from 0 to 10 million tokens.
• Demonstrates significant performance degradation in models like GPT-4 beyond 10% of their input capacity.
• Fine-tuning smaller models (e.g., RMT 137M, Mamba 130M) indicates task solvability.
• Recent leaderboard updates include Gemini 2.0, Llama-4-Scout, xLSTM, Gemma-3, Phi-4, and Llama-3.1 variants.

Maintenance & Community

The project was presented at NeurIPS 2024 and has a pre-print available on arXiv. It is a collaboration between AIRI, DeepPavlov.ai, and the London Institute for Mathematical Sciences. The community is invited to contribute by testing models, sharing insights, developing new tasks, and improving the benchmark via GitHub pull requests.

Licensing & Compatibility

The code is released under the Apache 2.0 License. The data utilizes PG19 corpora (Apache 2.0 License) and the bAbI dataset (BSD License). Apache 2.0 is generally permissive for commercial use and integration into closed-source projects.

Limitations & Caveats

Current state-of-the-art LLMs exhibit substantial performance degradation on BABILong tasks, particularly as context length increases significantly beyond their claimed capacities. Retrieval-Augmented Generation (RAG) methods have not shown improvements in this benchmark. The benchmark is designed to expose these limitations, indicating that robust long-context reasoning remains an open research problem.