The Anarchy LLM-VM project aims to provide an open-source, highly optimized backend for running Large Language Models (LLMs) locally. It targets developers and researchers seeking to accelerate AGI development, reduce costs, and gain flexibility by running various open-source LLMs with advanced features like tool usage, memory, and data augmentation.

How It Works

The LLM-VM acts as a virtual machine for human language, orchestrating data, models, prompts, and tools. It employs a multi-level optimization strategy, from agent-level coordination down to assembly code, incorporating techniques like state-of-the-art batching, sparse inference, quantization, distillation, and multi-level colocation. This approach aims to deliver high performance and efficiency for local LLM execution, supporting model and architecture agnosticism.

Quick Start & Requirements

• Installation: pip install llm-vm or clone the repository and run ./setup.sh (macOS/Linux) or .\windows_setup.ps1 (Windows).
• Prerequisites: Python >= 3.10. System requirements vary by model, with RAM being a common limiting factor (16GB recommended). OpenAI models require an LLM_VM_OPENAI_API_KEY environment variable.
• Links: anarchy.ai

Highlighted Details

• Supports tool usage via agents (FLAT, REBEL).
• Features inference optimization through batching, quantization, and distillation.
• Enables task auto-optimization via student-teacher distillation and data synthesis.
• Offers library callable Python interface and HTTP endpoints.

Maintenance & Community

• Development Status: DEVELOPMENT ON PAUSE.
• Community: Active Discord community for contributors.
• Contributors: Notable contributors include Matthew Mirman (CEO) and Victor Odede.

Licensing & Compatibility

• License: MIT License.
• Compatibility: Permissive for commercial use and closed-source linking.

Limitations & Caveats

The project is currently in BETA with development on pause. Several advanced features like live data augmentation, web playground, load-balancing, output templating, and persistent stateful memory are listed on the roadmap and not yet implemented.