Guidance is a Python programming language and paradigm for controlling large language models (LLMs), enabling structured output, reduced latency, and lower costs compared to traditional prompting or fine-tuning. It's designed for developers and researchers who need precise control over LLM generation, offering features like constrained generation, tool integration, and stateful control.

How It Works

Guidance programs are written in Python, leveraging f-strings for templating and a special gen function for LLM interaction. It allows interleaving generation with control flow (conditionals, loops) and tool use. Generation can be constrained using regular expressions, context-free grammars, or predefined select options. The library handles "token healing" to prevent issues with token boundaries, ensuring text-based control.

Quick Start & Requirements

• Install via pip: pip install guidance
• Supports various backends including Transformers, llama.cpp, OpenAI, AzureAI, and VertexAI.
• Requires Python. GPU acceleration is recommended for performance with local models.
• Official documentation and example notebooks are available.

Highlighted Details

• Constrained Generation: Enforce output structure with regex, CFGs, and select options.
• Tool Integration: Seamlessly call external tools during generation, with automatic argument capture and result injection.
• Backend Agnostic: Write a single Guidance program that can run across multiple LLM providers and local models.
• Stateful Control: Manage generation state and logic within Python functions, avoiding intermediate parsing steps.

Maintenance & Community

The project is actively maintained. Links to community resources like Discord or Slack are not explicitly mentioned in the README.

Licensing & Compatibility

The project is released under the MIT License, permitting commercial use and integration with closed-source applications.

Limitations & Caveats

While Guidance aims for broad backend compatibility, the most powerful control features (like complex grammars) are best supported with local backends like Transformers and llama.cpp. Remote API integrations may rely on "optimistic running," where complex constraints might fail if the model deviates from the expected output.