RAG-FiT is a modular Python framework designed to enhance Large Language Models (LLMs) for Retrieval Augmented Generation (RAG) tasks through fine-tuning. It targets researchers and developers looking to improve RAG performance by creating specialized RAG-augmented datasets, efficiently training models using Parameter-Efficient Fine-Tuning (PEFT), and evaluating the improvements with RAG-specific metrics.

How It Works

RAG-FiT employs a four-module architecture: Dataset Creation, Training, Inference, and Evaluation. The Dataset Creation module handles data processing, retrieval integration, and prompt templating to generate RAG-augmented datasets in a consistent format. The Training module leverages PEFT techniques and libraries like TRL for efficient model fine-tuning on these datasets. Inference generates predictions using fine-tuned or base models, and the Evaluation module assesses performance using a suite of RAG-aware metrics, including EM, F1, ROUGE, BERTScore, Deepeval, and RAGAS, which can utilize rich metadata beyond simple text.

Quick Start & Requirements

• Install: pip install -e .
• Optional Packages: pip install -e .[haystack] or pip install -e .[deepeval]
• Prerequisites: Python, standard ML libraries. Specific RAG backends (like Haystack) or evaluation tools (like Deepeval) require additional installs.
• Documentation: PubmedQA Tutorial (https://github.com/IntelLabs/RAG-FiT/blob/main/tutorials/pubmedqa/README.md)

Highlighted Details

• Supports PEFT for efficient fine-tuning.
• Modular design with configuration-as-code via Hydra for workflow customization.
• RAG-specific evaluation metrics that utilize retrieval results, reasoning, and citations.
• Facilitates fast prototyping and experimentation across the RAG pipeline.

Maintenance & Community

• Actively maintained by Intel Labs.
• Welcomes comments, suggestions, issues, and pull requests.
• Paper: RAG Foundry: A Framework for Enhancing LLMs for Retrieval Augmented Generation (arXiv:2408.02545).

Licensing & Compatibility

• License: Apache 2.0 License.
• Compatibility: Permissive license suitable for commercial use and integration with closed-source projects.

Limitations & Caveats

The framework is presented as a tool for enhancing LLMs for RAG tasks, implying that core LLM capabilities and RAG setup (retrievers, vector stores) are handled externally or through optional integrations. The effectiveness of fine-tuning is dependent on the quality and relevance of the generated RAG-augmented datasets.