<2-3 sentences summarising what the project addresses and solves, the target audience, and the benefit.> This Rust project, llm_client, simplifies integrating local Large Language Models (LLMs) by providing a robust interface to llama.cpp. It targets Rust developers seeking efficient, type-safe interaction with LLMs, automating the complex setup and management of the llama.cpp toolchain. The benefit is a streamlined development experience for leveraging local LLMs within Rust applications.

How It Works

The architecture uses LmcppToolChain for automated llama.cpp downloads, builds, and caching across platforms/backends (CPU, CUDA, Metal). LmcppServerLauncher spawns and monitors llama-server. LmcppServer provides a typed Rust API over fast Unix Domain Socket (UDS) IPC. This design prioritizes developer experience and performance for local LLM inference, offering type safety and potentially higher throughput than HTTP.

Quick Start & Requirements

Run CLI tools with cargo run --bin lmcpp-server-cli -- --webui. Specify models via Hugging Face URL (-u <URL>) or local path (-l /path/to/model.gguf). Requires a Rust toolchain. Automatically manages llama.cpp dependencies, supporting Linux, macOS, and Windows with CPU, CUDA, and Metal acceleration.

Highlighted Details

• Fully managed llama.cpp toolchain: automated downloads, builds, caching.
• Supports multiple llama.cpp versions and backends (CPU, CUDA, Metal) across platforms.
• Leverages fast Unix Domain Socket (UDS) IPC.
• Fully typed Rust API with documented server arguments and endpoints.
• Includes CLI tools for management and an optional Web UI.

Maintenance & Community

Maintained by Shelby Jenkins, contact via email/LinkedIn. The project has refocused on llama.cpp integration, with previous related crates spun off. No specific community channels (Discord/Slack) are listed.

Licensing & Compatibility

The README does not explicitly state the project's license. This omission requires further investigation for compatibility, especially for commercial use or integration into closed-source projects.

Limitations & Caveats

The project has intentionally narrowed its scope to be the definitive llama.cpp integration in Rust, moving away from broader AI agent DSLs. Previous related crates have been deprecated and moved to separate repositories. The absence of a declared license is a significant caveat for adoption.