AIKit is a comprehensive platform for easily hosting, deploying, building, and fine-tuning large language models (LLMs). It targets developers and researchers seeking a streamlined experience for LLM operations, offering an OpenAI-compatible API for broad client integration and support for various model formats and hardware.

How It Works

AIKit leverages LocalAI for inference, providing an OpenAI API-compatible REST endpoint that simplifies integration with existing tools. For fine-tuning, it integrates Unsloth, enabling fast and memory-efficient model customization. The platform is containerized using Docker, offering minimal image sizes and broad compatibility across CPU architectures (AMD64, ARM64) and NVIDIA GPUs, with experimental support for Apple Silicon.

Quick Start & Requirements

• Install/Run: docker run -d --rm -p 8080:8080 ghcr.io/sozercan/llama3.1:8b
• Prerequisites: Docker. NVIDIA GPU and CUDA Toolkit required for GPU acceleration.
• Setup: Minimal setup time, primarily dependent on Docker installation and model download.
• Docs: AIKit website

Highlighted Details

• OpenAI API compatible inference endpoint.
• Supports GGUF, GPTQ, EXL2, GGML, Mamba, and Mixtral models.
• Fine-tuning capabilities via Unsloth.
• Multi-modal and image generation support.
• AMD64, ARM64, NVIDIA GPU, and experimental Apple Silicon acceleration.
• Air-gapped environment support.

Maintenance & Community

The project is actively maintained by sozercan. Further community and roadmap details are available via the AIKit website.

Licensing & Compatibility

Licenses vary by model: Llama (Llama), Mixtral (Apache 2.0), Phi 3.5 (MIT), Gemma 2 (Gemma), Codestral (MNLP), QwQ (Apache 2.0), Flux 1 Dev (FLUX.1 [dev] Non-Commercial License). Compatibility for commercial use depends on the specific model's license.

Limitations & Caveats

Apple Silicon support is experimental and limited to GGUF models. The Flux 1 Dev model has a non-commercial license restriction.