This repository is a curated list of open-source tools and resources for running Artificial Intelligence models, particularly Large Language Models (LLMs), locally on user hardware. It targets developers, researchers, and enthusiasts seeking to leverage AI without relying on cloud services, offering a comprehensive overview of inference engines, UIs, fine-tuning tools, and agent frameworks.

How It Works

The project acts as a directory, categorizing and detailing various local AI solutions. It highlights inference engines like llama.cpp (C/C++), vLLM (Python), and TensorRT-LLM (NVIDIA GPUs), along with user interfaces such as Oobabooga and LM Studio. The collection also includes frameworks for building AI applications (LangChain, LlamaIndex), agent orchestration (CrewAI, Auto-GPT), and model training/optimization (DeepSpeed, PEFT).

Quick Start & Requirements

This is a curated list, not a single installable tool. Individual projects within the list have their own setup requirements, often including Python, specific libraries (e.g., PyTorch, Transformers), and potentially GPU acceleration with CUDA. Links to specific project documentation are provided within the README for detailed setup.

Highlighted Details

• Comprehensive table comparing inference engines by model format, CPU/GPU support, UI availability, and language.
• Lists popular UIs for interacting with local LLMs, including Gradio-based and standalone applications.
• Covers developer tools and frameworks for building LLM-powered applications and agents.
• Includes resources on LLM training, optimization, and evaluation benchmarks.

Maintenance & Community

The repository is community-driven, with contributions welcomed. It lists several related communities and projects, including LocalLLaMA, singularity, StableDiffusion, and Hugging Face.

Licensing & Compatibility

The repository itself is a list and does not have a specific license. However, the individual tools and projects linked within it are distributed under various open-source licenses (e.g., MIT, Apache 2.0, GPL). Users must consult the licenses of each specific tool for compatibility and usage restrictions.

Limitations & Caveats

As a curated list, the project does not provide direct functionality. Users must evaluate and set up each individual tool, which can involve complex dependencies and hardware requirements, particularly for GPU-accelerated inference. The rapidly evolving nature of local AI means some listed tools may become outdated.