LARS is an open-source desktop application for running Large Language Models (LLMs) locally, designed for users who want to ground LLM responses in their own documents with detailed citations. It aims to be the ultimate RAG-centric LLM application, providing advanced referencing capabilities and a comprehensive document reader within the response window.

How It Works

LARS utilizes a pure llama.cpp backend for LLM inference, avoiding complex Python bindings or frameworks for direct control and performance. It implements Retrieval Augmented Generation (RAG) by embedding user documents into a vector database using various sentence-transformer models. When a query is made, relevant document chunks are retrieved and used to augment the LLM's context, enabling it to generate responses grounded in the provided data, complete with detailed citations including document names, page numbers, and highlighted text.

Quick Start & Requirements

• Install: Clone the repository (git clone https://github.com/abgulati/LARS), navigate into the directory, and install Python dependencies (pip install -r .\requirements.txt or pip3 install -r ./requirements.txt).
• Prerequisites: Python v3.10.x or above, PyTorch (with CUDA/ROCm if using GPU), llama.cpp build tools (C++ compiler, CMake), Nvidia CUDA Toolkit v12.2+ (for GPU acceleration), LibreOffice (for extended document format support), Poppler (for PDF processing).
• Run: cd web_app then python app.py (or python3 app.py). Navigate to http://localhost:5000/.
• First Run: Downloads a default LLM (Phi-3-Mini-Instruct-44) and an embedding model (all-mpnet-base-v2). Users must then place their own LLMs (GGUF format recommended) into the models directory and select them in the Settings.
• Docs: LARS Feature-Demonstration Video

Highlighted Details

• Supports a vast array of file formats including PDFs, Word docs, Excel, PowerPoint, and images.
• Offers advanced citation features with document name, page number, text highlighting, and image extraction.
• Provides four local embedding models (sentence-transformers/all-mpnet-base-v2, BGE-Base, BGE-Large) and one cloud-based (Azure Text-Ada).
• GPU-accelerated inferencing via Nvidia CUDA is supported, with extensive configuration options for LLM settings.
• Includes three text extraction methods: local, Azure ComputerVision OCR (with a free tier), and Azure AI Document Intelligence OCR for tables.

Maintenance & Community

The project is actively developed by abgulati. Support and donations are encouraged via PayPal. The development roadmap indicates ongoing work on ease-of-use features, performance enhancements, and research into new RAG techniques and local OCR with Vision LLMs.

Licensing & Compatibility

The repository does not explicitly state a license in the README. Compatibility for commercial use or closed-source linking is not specified.

Limitations & Caveats

Azure OCR options incur API costs and are not bundled. Some Azure libraries are not available on macOS. Users must manually manage LLM downloads and selection. The README warns against tweaking settings or submitting queries while a response is being generated.