Summary

This project addresses the complexity of deploying large language models (LLMs) locally, particularly on multi-GPU systems and with Mixture-of-Experts (MoE) architectures. It automates the configuration of llama.cpp, determining optimal flags, tensor splits, and MoE expert placement. This offers an Ollama alternative for users with advanced hardware, enabling larger models to run efficiently across VRAM and system RAM.

How It Works

ggrun automates llama.cpp configuration by analyzing hardware (GPU, RAM, PCIe layout) to compute optimal --tensor-split and -ot flags based on GGUF tensor sizes. This enables distribution of large MoE models across VRAM and system RAM. It also automatically detects and validates multimodal projectors for seamless vision model integration. The tool recommends and downloads suitable model quantizations from Hugging Face, tailored to available VRAM.

Quick Start & Requirements

• Installation:
  • Linux/macOS: curl -fsSL https://raw.githubusercontent.com/raketenkater/ggrun/main/setup.sh | bash
  • Windows (PowerShell): iwr -useb https://raw.githubusercontent.com/raketenkater/ggrun/main/install.ps1 | iex
• Prerequisites: Linux CUDA builds compile from source for specific GPUs. Prebuilt bundles are available.
• Links: Installation and usage details are available in docs/install.md and docs/usage.md.

Highlighted Details

• Performance Benchmarks: Demonstrates significant speedups over llama.cpp --fit and Ollama for various models, especially MoE architectures, on multi-GPU setups.
• AI Tune: Automatically benchmarks flag sets to cache the fastest configuration.
• Hardware-Matched Downloads: Ranks models and downloads GGUF quantizations sized for available VRAM.
• Multimodal Support: Automatic detection and validation of matching mmproj for vision models.
• Crash Recovery: Mentioned as a feature for robustness.

Maintenance & Community

No specific details on contributors, sponsorships, community channels, or roadmaps were found in the provided README snippet.

Licensing & Compatibility

• License: MIT
• Compatibility: The MIT license permits commercial use and linking with closed-source projects.

Limitations & Caveats

• Backend Support: AMD and Intel GPUs utilize Vulkan; ROCm/HIP are not supported. Linux CUDA builds require source compilation.
• Development Status: Implied ongoing development with support for "fork backends" and "brand-new architectures."