This project provides an offline, privacy-focused solution for interacting with local documents via a chat interface. It targets users who want to avoid cloud dependencies and leverage local Large Language Models (LLMs) for document analysis and querying.

How It Works

The system utilizes a Retrieval Augmented Generation (RAG) pipeline. Documents are processed, chunked, and embedded using nomic-embed-text via Ollama. These embeddings are stored in a persistent Chroma vector database for efficient retrieval. User queries are also embedded, and relevant document chunks are fetched from the database to provide context to the local LLM (e.g., llama3:8b) for response generation.

Quick Start & Requirements

• Install Ollama from https://ollama.ai/download.
• Clone the repository and install dependencies: pip install -r requirements.txt.
• Pull required Ollama models: ollama pull nomic-embed-text and ollama pull llama3:8b.
• Start Ollama server: ollama serve.
• Run the Streamlit app: python -m streamlit run local_llama_v3.py.

Highlighted Details

• Leverages Ollama for local LLM integration and nomic-embed-text for embeddings.
• Supports PDF, TXT, DOCX, and MD file formats.
• Utilizes Haystack 2.0 for the RAG framework.
• Employs a persistent Chroma vector database for reusable document indexing.

Maintenance & Community

The project acknowledges the Ollama team, Haystack, and The-Bloke. No specific community links or roadmap details are provided in the README.

Licensing & Compatibility

Licensed under the Apache 2.0 License. This license is permissive and generally compatible with commercial use and closed-source linking.

Limitations & Caveats

The project relies on external tools like Ollama and Streamlit, which have their own system requirements. Performance will be dependent on the user's local hardware and the chosen LLM.