Kolosal AI is an open-source desktop application for running large language models (LLMs) offline on personal devices. It targets users seeking a lightweight, privacy-focused alternative to cloud-based AI services, enabling local inference and custom model training on a wide range of hardware.

How It Works

Kolosal AI is built using C++17 and CMake, compiled into a compact ~20 MB executable. It leverages the Genta Personal Engine, which is based on llama.cpp, to support various LLMs like Mistral, LLaMA, and Qwen. The application is designed for universal hardware compatibility, running on CPUs with AVX2 instructions and supporting AMD and NVIDIA GPUs, with an optional Vulkan backend for GPU acceleration.

Quick Start & Requirements

• Install: Clone the repository, ensure dependencies are met, configure with CMake (e.g., cmake -S .. -B . -DCMAKE_BUILD_TYPE=Release), and build (e.g., cmake --build . --config Release).
• Prerequisites: CMake 3.14+, C++17 compiler (MSVC, GCC 7+), OpenSSL, CURL. Vulkan SDK is optional for Vulkan backend.
• Setup: Requires manual download/placement of OpenSSL and CURL binaries if not system-installed.
• Docs: https://github.com/Genta-Technology/Kolosal

Highlighted Details

• Lightweight (~20 MB compiled) and portable, suitable for edge devices.
• Universal hardware support: AVX2 CPUs, AMD & NVIDIA GPUs.
• Powered by Genta Personal Engine (based on llama.cpp).
• Supports popular LLMs (Mistral, LLaMA, Qwen).
• Facilitates local dataset generation and model training.

Maintenance & Community

• Developed by Genta Technology, a student team focused on on-premise and on-edge AI.
• Community support via Discord.
• https://github.com/Genta-Technology/Kolosal

Licensing & Compatibility

• License: Apache 2.0.
• Compatible with commercial use and closed-source linking.
• External dependencies use their own licenses (MIT, zlib, Public Domain).

Limitations & Caveats

The build process requires manual management of external dependencies like OpenSSL and CURL if not system-installed. The Windows-specific resource file (resource.rc) may require modification for Linux/macOS builds.