LMQL is a Python-based programming language designed for efficient and constraint-guided interaction with Large Language Models (LLMs). It allows developers to seamlessly integrate LLM calls within traditional Python code, offering fine-grained control over model output through a declarative constraint language and advanced decoding strategies.

How It Works

LMQL treats top-level strings as LLM queries, enabling direct integration of LLM interactions into Python programs. It supports features like multi-variable templates, conditional distributions, and data type constraints via logit masking. The language leverages speculative execution and tree-based caching for optimized runtime performance, supporting synchronous and asynchronous APIs for parallel query execution.

Quick Start & Requirements

• Install via pip: pip install lmql
• For local GPU support (PyTorch >= 1.11): pip install lmql[hf]
• Running the playground IDE requires Node.js.
• OpenAI API key configuration is necessary for using OpenAI models.
• Tested OS: Ubuntu 22.04 (CUDA 12.0), Windows 10 via WSL2 (CUDA 11.7) for GPU; Ubuntu 22.04, macOS 13.2, Windows 10 via WSL2 for non-GPU.
• Documentation: Explore Examples, Playground IDE

Highlighted Details

• Python syntax integration for LLM queries.
• Advanced decoding techniques (beam search, best_k).
• Logit masking for output constraints (token length, data types, stopping phrases).
• Optimized runtime with speculative execution and constraint short-circuiting.
• Multi-model support (OpenAI, Azure OpenAI, Hugging Face Transformers).
• Integrates with LangChain and LlamaIndex.
• Offers an interactive Playground IDE and VS Code Extension.

Maintenance & Community

• Community-centric project with contribution guidelines.
• Discord server available for community interaction.

Licensing & Compatibility

• License: MIT.
• Compatible with commercial use and closed-source linking.

Limitations & Caveats

The bleeding-edge development version installed via pip install git+https://github.com/eth-sri/lmql is not continuously tested and may be less stable. The non-GPU version only supports API-integrated models.