RKLLama provides an Ollama-like server and client for running Large Language Models (LLMs) on Rockchip RK3588/3576 NPUs. It targets developers and users of these embedded systems, enabling efficient on-device AI inference with NPU acceleration.

How It Works

RKLLama leverages the rkllm-runtime library (V 1.2.1b1) to execute LLMs directly on the Rockchip NPU, offering a significant performance advantage over CPU-only inference. It features a REST API server (app.py) and a command-line client (client.py) for model interaction, supporting partial Ollama API compatibility for seamless integration with existing tools.

Quick Start & Requirements

• Installation: Clone the repository and run ./setup.sh for standard installation, or use docker pull ghcr.io/notpunchnox/rkllama:main and docker run --privileged -p 8080:8080 ghcr.io/notpunchnox/rkllama:main for Docker.
• Hardware: Rockchip RK3588(S) or RK3576 devices (e.g., Orange Pi 5 series) with 16GB RAM recommended.
• OS: Ubuntu 24.04 arm64 or Armbian Linux 6.1.99-vendor-rk35xx.
• Dependencies: Python 3.8-3.12.
• Docs: API REST Documentation

Highlighted Details

• NPU acceleration for LLM inference on Rockchip platforms.
• Partial Ollama API compatibility ( /api/chat, /api/generate ).
• Full support for tool/function calling with multiple LLM formats.
• Direct model pulling from Hugging Face with simplified naming.

Maintenance & Community

Active development with contributions from ichlaffterlalu, TomJacobsUK, and Yoann Vanitou. Upcoming features include OpenAI API compatibility and multimodal model support.

Licensing & Compatibility

The repository does not explicitly state a license. Compatibility for commercial use or closed-source linking is not specified.

Limitations & Caveats

The project is at version 0.0.42 and lists several upcoming features, indicating ongoing development. Ollama API compatibility is partial, and conversion tools for GGUF/HF models are noted as "coming soon."