This repository provides "Clownfish," a method for constrained decoding in Large Language Models (LLMs) to enforce adherence to JSON schemas. It's designed for developers and researchers building LLM-powered applications that require reliable, structured output, such as generating API calls, SQL queries, or configuration files, thereby preventing hallucinations and ensuring data integrity.

How It Works

Clownfish implements "ControLogits," a novel approach that modifies the LLM's token generation process. It integrates custom "LogitProcessors" that, for each token prediction, evaluate candidate tokens against a streaming JSON schema parser. Invalid tokens are zeroed out, forcing the LLM to select only valid continuations according to the schema. This method leverages immutable data structures for efficient backtracking and schema validation, offering a more robust alternative to fine-tuning or prompt engineering alone.

Quick Start & Requirements

• Install via pip install -e . (from source).
• Requires Python 3.7+, PyTorch, Transformers, Pydantic.
• Example usage involves defining Pydantic models and calling create or create_api functions.
• See examples for detailed usage.

Highlighted Details

• Enables LLMs to generate strictly valid JSON output based on provided schemas.
• Addresses issues like hallucinated actions or code by enforcing structural constraints.
• Offers a third approach to LLM alignment beyond fine-tuning and prompt iteration.
• Demonstrates potential for perplexity to indicate confidence in schema adherence.

Maintenance & Community

• Primarily a personal project by Ben Newhouse (@newhouseb on Twitter).
• Open to feedback and contributions.

Licensing & Compatibility

• Licensed under the MIT License.
• Permissive for commercial use and integration into closed-source applications.

Limitations & Caveats

The README notes that OpenAI models do not expose all logits, limiting direct application to GPT-3.5/4 without workarounds. The current implementation might be inefficient, especially with minimal prompting, as the model may explore a large state space to find valid tokens.