This project provides Node.js bindings for llama.cpp, enabling developers to run large language models (LLMs) locally on their machines. It targets Node.js developers seeking to integrate AI capabilities into their applications, offering features like JSON schema enforcement for model output and automatic hardware adaptation.

How It Works

The library leverages pre-built binaries for common operating systems (macOS, Linux, Windows), falling back to building llama.cpp from source using CMake when necessary. This approach simplifies setup and ensures broad compatibility. It supports hardware acceleration via Metal, CUDA, and Vulkan, automatically adapting to the available hardware without manual configuration.

Quick Start & Requirements

• Install via npm: npm install node-llama-cpp
• Pre-built binaries are provided; fallback to CMake build if unavailable.
• Supports Metal, CUDA, Vulkan acceleration.
• Official documentation: Getting started guide, API reference
• Try without installing: npx -y node-llama-cpp chat

Highlighted Details

• Full TypeScript support for enhanced developer experience.
• CLI for direct model interaction without coding.
• Function calling capabilities for models.
• Embedding and reranking support.
• Protection against special token injection attacks.

Maintenance & Community

The project is actively maintained and up-to-date with the latest llama.cpp releases. Community contributions are welcomed via a dedicated guide.

Licensing & Compatibility

The project is released under the MIT license, permitting commercial use and integration into closed-source projects.

Limitations & Caveats

While pre-built binaries cover major platforms, users on less common architectures might need to rely on the CMake build process, which could introduce additional setup complexity.