Summary

Adaptive Chunking optimizes Retrieval Augmented Generation (RAG) pipelines by automatically selecting the best document chunking strategy. It evaluates multiple methods against intrinsic quality metrics, enhancing retrieval completeness and answer correctness for more robust AI applications.

How It Works

The framework dynamically chooses chunking strategies (Recursive, Page, LLM Regex, Semantic) per document. It employs five intrinsic quality metrics—Size Compliance (SC), Intrachunk Cohesion (ICC), Contextual Coherence (DCC), Block Integrity (BI), and Filtered Missing Reference Error (RC)—to score chunking outputs. This adaptive selection optimizes chunk granularity and semantic integrity, outperforming fixed strategies in RAG evaluations. The design is modular, allowing custom chunker and metric integration.

Quick Start & Requirements

Install via pip: pip install -e ".[dev]" for full setup. Requires Python 3.11+ and spaCy models (python -m spacy download en_core_web_sm). Optional dependencies include GPU (semantic chunking), OpenAI API keys (LLM regex, RAG eval), and Jina API keys (faster metrics). Paper details: arXiv (https://arxiv.org/abs/2603.25333), LREC 2026 (https://lrec2026.info/). Paper reproduction can be resource-intensive.

Highlighted Details

• Official implementation accepted at LREC 2026.
• Achieves superior RAG performance: 67.7% Retrieval Completeness (vs. ~59% baselines) and 78.0% Answer Correctness (vs. ~70-73% baselines).
• Intrinsic metrics yield a mean score of 91.07% for Adaptive Chunking, outperforming other methods.
• Supports multiple document parsing backends: Docling (default), PyMuPDF, Azure Document Intelligence, and Excel.

Maintenance & Community

The project is the official implementation of a LREC 2026 paper by Ekimetrics authors. While specific community channels are not detailed, the project actively works on replacing non-permissively licensed dependencies to broaden usability.

Licensing & Compatibility

The core library uses the MIT License. Optional components have different licenses: [coref] uses CC BY-NC-SA 4.0 (non-commercial, share-alike), and [parsing] extras include AGPL-3.0 or commercial licenses for pymupdf4llm. These may restrict commercial use or closed-source integration, though permissive alternatives are sought.

Limitations & Caveats

Optional features like coreference resolution and specific PDF parsers are subject to non-commercial or copyleft licenses. Semantic chunking requires a GPU and flash-attention. The RAG evaluation pipeline is computationally expensive, relying on external APIs and significant processing.