TinyChatEngine is an on-device inference library for Large Language Models (LLMs) and Visual Language Models (VLMs), targeting developers and researchers building edge AI applications. It enables real-time, private LLM/VLM deployment on laptops, cars, and robots by implementing advanced model compression techniques like SmoothQuant and AWQ.

How It Works

TinyChatEngine leverages SmoothQuant and AWQ for LLM compression, reducing model size and computational requirements. SmoothQuant addresses quantization difficulty by migrating it from activations to weights, while AWQ protects salient weight channels by analyzing activation magnitudes. The core inference engine is a from-scratch C/C++ implementation designed for universal compatibility across x86, ARM (including Apple Silicon), and NVIDIA GPUs, eliminating external library dependencies for a streamlined deployment.

Quick Start & Requirements

• Install: Clone the repository (git clone --recursive) and install Python dependencies (pip install -r requirements.txt).
• Prerequisites:
  • macOS: brew install boost llvm, Xcode for Metal compiler.
  • Windows (CPU): GCC compiler with MSYS2, pacman -S --needed base-devel mingw-w64-x86_64-toolchain make unzip git.
  • Windows (NVIDIA GPU - Experimental): CUDA Toolkit, Visual Studio with C/C++ support.
  • NVIDIA GPU: CUDA compute capability >= 6.1.
• Setup: Download quantized models using python tools/download_model.py. Compile with make chat -j.
• Docs: Model Zoo, VILA Demo

Highlighted Details

• Supports Llama-3, Llama-2, Code Llama, Mistral, VILA, LLaVA, and StarCoder models.
• Offers various quantization methods including FP32, W4A16, W4A32, W4A8, and W8A8.
• Optimized weight reordering (QM_ARM, QM_x86, QM_CUDA) for specific architectures to minimize runtime overhead.
• Awarded Best Paper at MLSys 2024 for AWQ and TinyChat.

Maintenance & Community

• Actively developed by MIT HAN Lab.
• Recent updates include Llama-3 support and VLM extensions.
• Related projects: TinyEngine, SmoothQuant, AWQ.

Licensing & Compatibility

• The README does not explicitly state a license. Compatibility for commercial use or closed-source linking is undetermined.

Limitations & Caveats

• Windows NVIDIA GPU support is experimental.
• Support for GPUs with compute capability < 6.1 is untested.
• The license is not specified, which may impact commercial adoption.