PICARD (Parsing Incrementally for Constrained Auto-Regressive Decoding) is a library for enforcing structural constraints during text generation, particularly useful for tasks like text-to-SQL. It enables unmodified pre-trained language models to generate valid, semantically correct outputs without requiring specialized decoder architectures or retraining.

How It Works

PICARD integrates with standard beam search decoding by incrementally parsing the generated token sequences. It uses an incremental parsing library (attoparsec) to check the validity of potential next tokens against predefined grammar rules (e.g., SQL syntax). Invalid tokens are pruned from the beam, ensuring that only syntactically correct sequences are explored. This approach avoids modifying the underlying language model and requires no additional training.

Quick Start & Requirements

• Installation: Clone the repository and initialize submodules:

  git clone git@github.com:ElementAI/picard.git
  cd picard
  git submodule update --init --recursive
  

• Prerequisites: Python, PyTorch, Hugging Face Transformers, attoparsec (via Haskell). Training configuration is optimized for GPUs with 40GB memory.
• Resources: Pre-trained models are available on Hugging Face (e.g., tscholak/cxmefzzi for T5-3B). Docker images are provided for development, training, and evaluation.
• Links: Paper, Video, Hugging Face Models.

Highlighted Details

• Achieves state-of-the-art performance on Spider and CoSQL text-to-SQL benchmarks using unmodified T5 models.
• Reduces SQL execution errors from 12% to 2% on the Spider development set when used with a T5-3B model.
• Works with character-, sub-word-, and word-level language models.
• Requires no model modification, special vocabulary, or additional training.

Maintenance & Community

This project originated from Element AI and is now a ServiceNow Research project. The primary contributor is Torsten Scholak.

Licensing & Compatibility

The repository does not explicitly state a license in the README. However, the project is hosted on GitHub under the ServiceNow organization. Users should verify licensing for commercial or closed-source use.

Limitations & Caveats

The README mentions that the default training configuration is optimized for a 40GB GPU, potentially limiting training on consumer hardware. The parsing rules are specific to a subset of SQLite syntax for the text-to-SQL task, and extending to other SQL dialects or grammars would require custom parser implementation.