This project provides a local, privacy-respecting AI assistant capable of voice recognition, natural language processing, and task execution. It targets users seeking an offline, customizable AI companion for daily tasks like summarization, rephrasing, and question answering, powered by various open-source LLMs.

How It Works

The assistant leverages llama.cpp and llama-cpp-python for efficient local inference of LLMs, supporting quantized models for reduced resource usage. It integrates whisper.cpp for offline speech-to-text and offers multimodal capabilities with models like Moondream2 and LLaVA. The architecture supports streaming responses and includes a global hotkey for quick access.

Quick Start & Requirements

• Install via pip: pip install llama-assistant
• Requires Python 3.10+.
• PortAudio installation is necessary for microphone access, with platform-specific instructions provided for macOS, Debian/Ubuntu, and Windows.
• Apple Silicon users can enable Metal support for llama-cpp-python for performance gains.
• Official documentation and website: https://llama-assistant.nrl.ai/

Highlighted Details

• Supports a range of text and multimodal models, including Llama 3.2, Qwen2.5, Gemma-2, DeepSeek-R1, Moondream2, MiniCPM-v2.6, and LLaVA.
• Features offline speech recognition via WhisperCPP.
• Includes a global hotkey (Cmd+Shift+Space on macOS) for quick activation.
• Customizable UI with adjustable transparency.

Maintenance & Community

The project is actively developed with a roadmap including features like wake word detection, a plugin system, and multi-language support. Contributions are welcome via pull requests.

Licensing & Compatibility

Licensed under GPLv3. This license may impose copyleft restrictions on derivative works, potentially requiring modifications to be open-sourced if distributed.

Limitations & Caveats

The project is described as a work in progress, with several features marked as TODO, including a plugin system, email integration, and multi-language support. The current implementation may not be suitable for production environments without further development and testing.