Amis-LocalAI enables running large open-weight language models locally on consumer hardware with limited RAM. It provides a local-first workspace for private agents and models by treating fast NVMe storage as a managed memory extension. This allows users to deploy significantly larger models with reduced memory footprints and improved performance, targeting developers and power users focused on local AI inference.

How It Works

The core innovation is AI Storage Expansion, leveraging fast NVMe SSDs as a managed DRAM extension to increase addressable model weight space. This is coupled with MoE-aware sparse activation (processing only relevant experts per token), page-aware memory mapping for prefetching, and CPU-first scheduling with opportunistic offload to iGPUs/dGPUs/NPUs. This runtime technique dramatically reduces memory pressure and enhances inference speed on consumer hardware.

Quick Start & Requirements

Setup currently requires cloning the repository and building the preview runtime from source; standalone installers and an OpenAI-compatible endpoint are planned. The primary requirement is a GGUF-quantized model; the runtime defaults to CPU-first execution with optional hardware acceleration. Links to the repository, roadmap, and benchmarking guide are provided.

Highlighted Details

• AI Storage Expansion: Treats fast NVMe storage as a managed DRAM extension, allowing models to exceed installed RAM.
• Memory Reduction & Speed: Achieves ~50% RAM reduction (e.g., 35B MoE model from 21.8 GB to 10.8 GB) with up to +61% faster prefill vs. baseline llama.cpp.
• Consumer Hardware Focus: Optimized for devices like MacBooks, Mini PCs, and AI PCs with NPU/CPU silicon, not datacenter GPUs.
• MoE-Aware Sparse Activation: Activates only necessary model experts per token, reducing computation and memory access.

Maintenance & Community

Maintained by "The Amis-LocalAI Authors," community interaction occurs via GitHub Discussions/Issues. Discord/X channels are planned. Contact is available for commercial inquiries/partnerships. A public roadmap details development progress.

Licensing & Compatibility

Released under GNU AGPL-3.0, a strong copyleft license. Commercial licensing or OEM integration requires explicit contact with maintainers at contact@cpilot.net.

Limitations & Caveats

The runtime is in preview; only selected models/devices are fully validated, others are exploratory. Broader hardware support and a standalone runtime release are still under development.