The RAG Experiment Accelerator is a tool for researchers, data scientists, and developers to streamline the experimentation and evaluation of Retrieval-Augmented Generation (RAG) systems, specifically leveraging Azure AI Search and Azure OpenAI. It automates the process of testing various search parameters, query strategies, and response quality, generating detailed reports and visualizations to identify optimal configurations.

How It Works

This tool employs a config-driven approach, allowing users to define experiments by specifying ranges for search engine parameters, query sets, and evaluation metrics. It integrates with Azure AI Search, Azure Machine Learning, and Azure OpenAI, supporting multiple search types (text, vector, hybrid) and advanced features like sub-querying and LLM-based re-ranking. A custom document loader enhances data ingestion by intelligently formatting tables and excluding irrelevant content.

Quick Start & Requirements

• Installation: Clone the repository and run within a development container (requires WSL) or install locally using pip install . after setting up a conda environment.
• Prerequisites: Azure subscription, Azure OpenAI Service access, Azure CLI, Docker Desktop (for dev container), Python 3.11.
• Setup: Requires provisioning Azure infrastructure via Azure Developer CLI (azd provision) or Azure CLI. Configuration involves updating .env and config.json files.
• Documentation: Overview

Highlighted Details

• Supports multiple document loaders, including a custom Azure Document Intelligence loader with advanced text processing.
• Offers diverse search types: pure text, pure vector, cross-vector, multi-vector, and hybrid.
• Includes LLM-based re-ranking and component-wise evaluation metrics (e.g., context recall, answer relevance).
• Features data sampling for faster experimentation on large datasets, with clustering for representative subsets.

Maintenance & Community

This is a Microsoft-maintained project. Contributions are welcome via pull requests after signing a Contributor License Agreement.

Licensing & Compatibility

The project is licensed under the MIT License, permitting commercial use and integration with closed-source projects.

Limitations & Caveats

The solution has been tested with GPT 3.5 Turbo; compatibility with other Azure OpenAI models requires further testing. Sampling is currently only supported locally, not on distributed AML compute clusters. Some evaluation metrics are not semantic-aware, and LLM-based metrics are non-deterministic.