This library enforces structured output formats (JSON Schema, Regex) for language models, targeting developers and researchers needing reliable, predictable LLM responses. It enhances LLM usability by filtering token generation at each step, ensuring adherence to specified formats while minimizing constraints on the model's creative freedom.

How It Works

The core mechanism combines a character-level parser with a tokenizer's prefix tree. The character-level parser defines the valid sequence of characters for a given format (e.g., JSON Schema, Regex). Simultaneously, a prefix tree represents all possible token sequences the LLM can generate. By intersecting these two structures, the library identifies valid tokens that advance both the format parsing and the LLM's generation, effectively filtering out invalid token choices at each generation step. This approach allows the LLM to control whitespace and field ordering, potentially improving output quality.

Quick Start & Requirements

• Install via pip: pip install lm-format-enforcer
• Requires Python 3.x.
• For GPU usage with Hugging Face Transformers, install transformers, torch, huggingface_hub, optimum, and auto-gptq (with CUDA 11.8 specified).
• Official Colab Notebook: https://colab.research.google.com/github/noamgat/lm-format-enforcer/blob/main/docs/notebooks/transformers_example.ipynb

Highlighted Details

• Supports integration with Transformers, LangChain, LlamaIndex, llama.cpp, vLLM, Haystack, and TensorRT-LLM.
• Handles JSON Schema (including optional fields, nested structures, and recursive classes), JSON Mode, and Regular Expressions.
• Supports batched generation and beam searches.
• Allows LLMs to control JSON whitespace and field ordering, reducing potential hallucinations.
• Offers detailed diagnostics to analyze the impact of format enforcement on token selection.

Maintenance & Community

• Active development with contributions from multiple individuals.
• Integration into vLLM server noted.
• Community support channels are not explicitly mentioned in the README.

Licensing & Compatibility

• The README does not explicitly state a license. Compatibility for commercial use or closed-source linking is not specified.

Limitations & Caveats

• Cannot be used with API-based solutions like OpenAI ChatGPT due to the requirement for direct logit access.
• Regex syntax support is not 100% complete due to reliance on interegular.
• The Regex parser can only generate characters present in the tokenizer's vocabulary.