picard  by ServiceNow

PICARD: Constrained decoding research paper for language models

created 3 years ago
362 stars

Top 78.7% on sourcepulse

GitHubView on GitHub
Project Summary

PICARD (Parsing Incrementally for Constrained Auto-Regressive Decoding) is a library for enforcing structural constraints during text generation, particularly useful for tasks like text-to-SQL. It enables unmodified pre-trained language models to generate valid, semantically correct outputs without requiring specialized decoder architectures or retraining.

How It Works

PICARD integrates with standard beam search decoding by incrementally parsing the generated token sequences. It uses an incremental parsing library (attoparsec) to check the validity of potential next tokens against predefined grammar rules (e.g., SQL syntax). Invalid tokens are pruned from the beam, ensuring that only syntactically correct sequences are explored. This approach avoids modifying the underlying language model and requires no additional training.

Quick Start & Requirements

  • Installation: Clone the repository and initialize submodules:
    git clone git@github.com:ElementAI/picard.git
    cd picard
    git submodule update --init --recursive
    
  • Prerequisites: Python, PyTorch, Hugging Face Transformers, attoparsec (via Haskell). Training configuration is optimized for GPUs with 40GB memory.
  • Resources: Pre-trained models are available on Hugging Face (e.g., tscholak/cxmefzzi for T5-3B). Docker images are provided for development, training, and evaluation.
  • Links: Paper, Video, Hugging Face Models.

Highlighted Details

  • Achieves state-of-the-art performance on Spider and CoSQL text-to-SQL benchmarks using unmodified T5 models.
  • Reduces SQL execution errors from 12% to 2% on the Spider development set when used with a T5-3B model.
  • Works with character-, sub-word-, and word-level language models.
  • Requires no model modification, special vocabulary, or additional training.

Maintenance & Community

This project originated from Element AI and is now a ServiceNow Research project. The primary contributor is Torsten Scholak.

Licensing & Compatibility

The repository does not explicitly state a license in the README. However, the project is hosted on GitHub under the ServiceNow organization. Users should verify licensing for commercial or closed-source use.

Limitations & Caveats

The README mentions that the default training configuration is optimized for a 40GB GPU, potentially limiting training on consumer hardware. The parsing rules are specific to a subset of SQLite syntax for the text-to-SQL task, and extending to other SQL dialects or grammars would require custom parser implementation.

Health Check
Last commit

1 year ago

Responsiveness

1 day

Pull Requests (30d)
0
Issues (30d)
0
Star History
9 stars in the last 90 days

Explore Similar Projects

Starred by Chip Huyen Chip Huyen(Author of AI Engineering, Designing Machine Learning Systems), Omar Sanseviero Omar Sanseviero(DevRel at Google DeepMind), and
4 more.

open_flamingo by mlfoundations

0.1%
4k
Open-source framework for training large multimodal models
created 2 years ago
updated 11 months ago
Feedback? Help us improve.