Ratchet is a cross-platform machine learning framework designed for web-first deployment, enabling GPU-accelerated inference in browsers and native applications. It targets developers seeking to integrate performant AI into existing production environments, offering a toolkit focused on inference, WebGPU/CPU execution, quantization, lazy computation, and in-place operations.

How It Works

Ratchet leverages WebGPU for hardware-accelerated computation, providing a unified API for both browser and native environments. Its design prioritizes efficient inference through first-class quantization support and lazy computation, minimizing overhead and maximizing performance on diverse hardware.

Quick Start & Requirements

• Install/Run: Experience via Hugging Face Spaces (Whisper, Phi).
• Prerequisites: Web browser with WebGPU support. JavaScript API demonstrated. Rust crate and CLI are forthcoming.
• Resources: Demo sites are available for immediate testing.

Highlighted Details

• Supports Whisper, Phi 2 & 3, and Moondream models, with Gemini 2 2B upcoming.
• Features asynchronous loading and caching via IndexedDB for web applications.
• Emphasizes quantization (e.g., Q8) for performance optimization.

Maintenance & Community

• Currently in active development, seeking community contributions.
• Community channels include Discord. Roadmap is available.

Licensing & Compatibility

• License is not explicitly stated in the README.

Limitations & Caveats

• The project is in active development, with ongoing work on the engine, model support, and compatibility. A Rust crate and CLI are not yet released.