CRAG  by HuskyInSalt

RAG method for improved generation robustness

created 1 year ago
398 stars

Top 73.7% on sourcepulse

GitHubView on GitHub
Project Summary

This repository provides the source code for Corrective Retrieval Augmented Generation (CRAG), a method to enhance the robustness of Large Language Models (LLMs) against inaccurate or irrelevant retrieved documents in Retrieval-Augmented Generation (RAG) systems. It is designed for researchers and developers working with LLMs and RAG to improve generation quality and reduce hallucinations.

How It Works

CRAG introduces a lightweight retrieval evaluator that assesses the quality of retrieved documents and assigns a confidence score. This score dynamically triggers different knowledge retrieval actions, including internal document decomposition and external web searches. A decompose-then-recompose algorithm further refines retrieved documents by focusing on key information and filtering out irrelevant content, making CRAG a plug-and-play module for existing RAG approaches.

Quick Start & Requirements

  • Installation: Clone the repository and create a Conda environment with Python 3.11. Install dependencies using pip install -r requirements.txt.
  • Prerequisites: Python 3.11, Conda, and an OpenAI API key and search API key (e.g., Google Search API) are required for external knowledge preparation.
  • Resources: Requires downloading pre-trained evaluator weights and fine-tuned LLaMA-2 models. Data preprocessing and knowledge preparation steps are also necessary.
  • Links: Official Paper

Highlighted Details

  • Implements a retrieval evaluator to assess document quality and trigger corrective actions.
  • Supports internal knowledge refinement via document decomposition (fixed_num, excerption, selection).
  • Integrates external knowledge gathering through question rewriting and web searching.
  • Offers a decompose-then-recompose algorithm to filter irrelevant information from retrieved documents.

Maintenance & Community

The project was last updated on October 8, 2024, with revisions to prompt formatting. Key releases include inference code, evaluator weights, and data preparation scripts. Further details on community channels or active development are not explicitly mentioned in the README.

Licensing & Compatibility

The repository does not explicitly state a license. The code is provided for research purposes, and citation of the associated paper is requested. Commercial use or linking with closed-source projects may require further clarification on licensing terms.

Limitations & Caveats

The evaluation results reported in the paper were based on the deprecated text-davinci-003 model, and may differ when using current models like gpt-3.5-turbo-instruct. The README also notes that the Self-CRAG inference and evaluator fine-tuning were planned for future release at the time of writing.

Health Check
Last commit

9 months ago

Responsiveness

Inactive

Pull Requests (30d)
0
Issues (30d)
1
Star History
36 stars in the last 90 days

Explore Similar Projects

Feedback? Help us improve.