GraphRAG-Bench addresses the effectiveness gap in Graph Retrieval-Augmented Generation (GraphRAG), where it often underperforms traditional RAG. It provides a comprehensive benchmark and dataset to identify scenarios where GraphRAG offers measurable benefits over vanilla RAG. Aimed at researchers and engineers, it facilitates systematic evaluation and offers practical application guidelines for GraphRAG systems.

How It Works

This project benchmarks GraphRAG by comparing its approach, which uses graphs to model hierarchical concept structures for enhanced knowledge retrieval, against traditional RAG. GraphRAG-Bench features a dataset with tasks of increasing difficulty (fact retrieval to creative generation) and a systematic evaluation pipeline covering graph construction, knowledge retrieval, and final generation. Its novelty lies in systematically investigating the conditions under which GraphRAG surpasses vanilla RAG and providing actionable insights.

Quick Start & Requirements

Installation involves pip install -r requirements.txt, with a strong recommendation for separate Conda environments per framework. An example setup for LightRAG is provided, requiring Python 3.10. Further details and framework-specific instructions are available in the Examples folder. Relevant links include the arXiv paper and Hugging Face datasets. Direct access to the project's leaderboard was not possible.

Highlighted Details

Features two domain-specific leaderboards (Literary/Fictional, Medical/Healthcare) with comprehensive metrics across Fact Retrieval, Complex Reasoning, Contextual Summarization, and Creative Generation. Supports DIGIMON for flexible benchmarking and has released LinearRAG, a relation-free method for efficient GraphRAG.

Maintenance & Community

Contributions are welcomed via email to GraphRAG@hotmail.com. Specific community channels or active maintainer information are not detailed in the README or accessible content.

Licensing & Compatibility

The project is licensed under the MIT license. Direct access to the LICENSE file was not possible, but the README and Hugging Face page indicate MIT. This license is generally permissive for commercial use and closed-source linking.

Limitations & Caveats

The benchmark acknowledges that GraphRAG frequently underperforms vanilla RAG on many real-world tasks, implying specific scenarios where it is less effective. The setup process emphasizes careful environment management due to potential dependency conflicts between different RAG frameworks. Direct access to the project's leaderboard and license details was not possible during this review.