This repository simplifies the installation and usage of Rockchip's Neural Processing Units (NPUs), specifically targeting the RK3588 and similar SoCs. It enables users to efficiently run AI models, including Large Language Models (LLMs) and computer vision tasks, directly on the NPU, offering a streamlined path for edge AI deployments.

How It Works

The project is architected around two core submodules: ezrknn-llm (based on rknn-llm v1.2.1) for LLM inference and ezrknn-toolkit2 (based on rknn-toolkit2 v2.3.2) for broader neural network applications. Models are converted into the .rkllm format optimized for Rockchip NPUs. This modular approach abstracts the complexities of the underlying Rockchip SDK, providing a unified interface for NPU acceleration. The ezrknn-llm submodule focuses on LLM conversion and runtime, supporting a wide range of models. The ezrknn-toolkit2 submodule provides easier installation for the RKNN Toolkit 2, enabling model conversion and inference for various neural network architectures.

Quick Start & Requirements

Installation is facilitated via a comprehensive script: curl https://raw.githubusercontent.com/Pelochus/ezrknpu/main/install.sh | sudo bash. Key requirements include:

• Hardware: Rockchip RK3588 or RK3588S SoC.
• Operating System: Debian 11, Ubuntu 22.04+, or derivatives. Armbian is recommended for its up-to-date NPU driver (v0.9.8+).
• NPU Driver: Version 0.9.6 or higher (verify with dmesg | grep -i rknpu).
• Python: Version 3.12. A detailed tutorial for setting up LLMs from scratch on an Orange Pi 5 Pro is available via an XDA article.

Highlighted Details

• Enables running various LLMs (e.g., Llama 2 7B, Qwen 1.8B) and models like YoloV5 on RK3588 NPUs.
• Provides utilities for monitoring NPU load, including rknputop and ntop.sh.
• Offers direct links to HuggingFace repositories with pre-converted .rkllm models.
• Includes demo videos showcasing LLMs operational on the NPU.
• The ezrknn-llm submodule supports numerous LLM architectures and provides guidance on fixing hallucinating models.
• The ezrknn-toolkit2 submodule includes examples for common models like ResNet18 and notes on NPU usage testing.

Maintenance & Community

The project is actively maintained by Pelochus, with a strong emphasis on community support through the r/RockchipNPU subreddit. This community is a valuable resource for troubleshooting, sharing development links, and discussing NPU-accelerated AI projects on Rockchip platforms. Contributions are welcomed for expanding SoC support, Android integration, LLM conversion, and documentation enhancement.

Licensing & Compatibility

The specific open-source license is not explicitly stated in the README. Compatibility is primarily focused on Linux environments running on RK3588/RK3588S hardware. Commercial use implications are undetermined due to the lack of explicit licensing information.

Limitations & Caveats

The project's scope is currently limited to RK3588 and RK3588S SoCs, excluding support for other Rockchip devices and Android. Users must ensure compatibility between LLM conversion versions and the rknn-llm runtime (e.g., models converted before v1.1.0 may not work with v1.1.x). Reliance on specific driver and Python versions may necessitate using recommended OS distributions like Armbian. The XDA tutorial notes that NPU-driven LLMs are still experimental and may exhibit quirks like self-talk or output truncation.

https://github.com/Pelochus/ezrknn-llm https://github.com/Pelochus/ezrknn-toolkit2 https://www.xda-developers.com/how-i-used-the-npu-on-my-orange-pi-5-pro-to-run-llms/ https://huggingface.co/imkebe https://huggingface.co/c01zaut https://huggingface.co/macrae https://huggingface.co/Pelochus/ezrkllm-collection https://www.reddit.com/r/RockchipNPU/comments/1ci7p72/rk3588_running_llama2_7b/ https://www.reddit.com/r/RockchipNPU/comments/1c0x7c2/first_llm_running_on_rk3588_npu/ https://www.reddit.com/r/RockchipNPU/