picoLLM is an on-device inference engine for large language models (LLMs), designed for high accuracy and privacy. It targets developers and researchers needing to run LLMs locally across various platforms, including desktops, mobile, and web browsers, with a key benefit being its efficient X-Bit quantization for reduced resource consumption.

How It Works

picoLLM utilizes a novel quantization algorithm called picoLLM Compression, which automatically optimizes bit allocation across LLM weights based on a task-specific cost function. This adaptive approach surpasses fixed-bit schemes like GPTQ, recovering significantly more accuracy (e.g., 91-100% MMLU score recovery at 2-4 bits for Llama-3-8b). Inference runs entirely locally, ensuring data privacy.

Quick Start & Requirements

• Install: pip3 install picollm (Python), dotnet add package PicoLLM (.NET), yarn add @picovoice/picollm-node (Node.js), etc.
• Prerequisites: An AccessKey from Picovoice Console is required for authentication, though inference is free for open-weight models. Model files must be downloaded separately from the Picovoice Console.
• Demos: Available for Python, .NET, Node.js, Android, iOS, Web, and C.
• Docs: https://picovoice.ai/docs/picollm/

Highlighted Details

• Supports a wide range of open-weight models including Gemma, Llama (2 & 3), Mistral, Mixtral, Phi-2, and Phi-3.
• Cross-platform compatibility: Linux, macOS, Windows, Raspberry Pi (4 & 5), Android, iOS, and major web browsers.
• Runs on both CPU and GPU.
• Offers an interrupt() function for halting generation.

Maintenance & Community

• Actively maintained by Picovoice. Recent releases include performance improvements and support for new models like Phi-3.5.
• No explicit community links (Discord/Slack) are provided in the README.

Licensing & Compatibility

• The SDKs and inference engine are free for open-weight models.
• An AccessKey is required, implying a licensing/authentication mechanism tied to Picovoice services, even for offline inference.

Limitations & Caveats

• An internet connection is required to validate the AccessKey, even though inference is offline.
• Model files must be downloaded separately from the Picovoice Console.