This project provides headless React Hooks for running Large Language Models (LLMs) directly in the browser using WebGPU, enabling privacy-preserving AI applications. It targets React developers seeking to integrate LLM capabilities client-side without server dependencies.

How It Works

The library leverages Apache TVM and MLC Relax, compiled for the browser via Emscripten, to run LLMs like Vicuna 7B. A SentencePiece tokenizer, also compiled via Emscripten, handles text processing. The model and runtime are loaded from a CDN and cached in browser storage. Off-the-main-thread WebWorkers execute the model, ensuring a responsive UI.

Quick Start & Requirements

• Install: npm install @react-llm/headless
• Requirements: WebGPU-enabled browser.
• Demo: cd packages/retro-ui && pnpm install && pnpm dev

Highlighted Details

• Supports Vicuna 7B models.
• Enables custom system prompts and role names.
• Offers completion options (max tokens, stop sequences).
• No data leaves the browser.
• Persistent conversation storage.
• Model caching for faster reloads.

Maintenance & Community

No specific contributors, sponsorships, or community links (Discord/Slack) are mentioned in the README.

Licensing & Compatibility

MIT License for the core library. The code under packages/headless/worker/lib/tvm is licensed under Apache 2.0. Compatible with commercial use and closed-source applications.

Limitations & Caveats

The README mentions Vicuna 7B, but the "How does it work?" section states Vicuna 13B. The project appears to be in early stages, with a focus on demonstrating core functionality.