CatAI provides a local AI assistant experience, enabling users to run GGUF models on their own computers with a chat UI and a simple Node.js API. It targets developers and power users seeking to leverage large language models offline, offering features like real-time streaming and fast model downloads.

How It Works

CatAI utilizes node-llama-cpp, a Node.js binding for llama.cpp, to run GGUF models efficiently. This approach allows for cross-platform compatibility (Windows, Linux, macOS) and leverages the performance optimizations of llama.cpp for local inference. The project also includes a CLI for model management and a web API for programmatic interaction.

Quick Start & Requirements

• Install globally: npm install -g catai
• Install a model: catai install meta-llama-3-8b-q4_k_m
• Start the server: catai up
• Requires Node.js.
• Supports multiple platforms including darwin-x64, linux-x64, win32-x64-msvc.
• Model downloads default to ~/catai.

Highlighted Details

• Auto-detects programming language.
• Real-time text streaming.
• Fast, multi-threaded model downloads.
• Offers a development API for programmatic interaction with models, including JSON schema grammar support.

Maintenance & Community

• Project is actively maintained.
• Contributions are welcome via a contributing guide.

Licensing & Compatibility

• MIT License for the CatAI package itself.
• Subject to the llama.cpp license, which is typically MIT.
• Compatible with commercial and closed-source applications.

Limitations & Caveats

The project relies on node-llama-cpp which is in beta. Specific platform support depends on the underlying llama.cpp build for node-llama-cpp.