This repository provides SimCTG, a contrastive framework for neural text generation, and its associated decoding method, contrastive search. It addresses the common issue of degenerate text generation (unnaturalness, repetition) in autoregressive models by calibrating the representation space and encouraging diversity while maintaining coherence. The framework is beneficial for researchers and practitioners in NLP seeking improved text generation quality.

How It Works

SimCTG introduces a contrastive training objective to regularize the model's representation space, aiming to make it more isotropic. This is complemented by contrastive search, a decoding strategy that balances local coherence with global diversity. By contrasting token probabilities against a contrastive baseline, it steers generation away from repetitive or nonsensical outputs, leading to more natural and coherent text.

Quick Start & Requirements

• Install via pip: pip install simctg --upgrade
• Requires PyTorch.
• Official Hugging Face integration and tutorials are available.
• Demo: https://huggingface.co/spaces/huggingface/contrastive_search
• Documentation: https://github.com/yxuansu/SimCTG/blob/main/docs/simctg.md

Highlighted Details

• Achieved state-of-the-art results on multiple benchmarks.
• Demonstrates strong performance across 16 languages, comparable to human text in some cases.
• Integrates seamlessly with Hugging Face Transformers for GPT-2, OPT, and other models.
• Supports various tasks including document generation, dialogue generation, and story generation.

Maintenance & Community

• Actively maintained with recent updates and integration into Hugging Face.
• NeurIPS'22 Spotlight paper.
• Follow-up work MAGIC released.
• Contact: ys484 at cam.ac.uk

Licensing & Compatibility

• The repository does not explicitly state a license. Users should verify licensing for commercial or closed-source use.

Limitations & Caveats

• The repository does not explicitly state a license, which may pose a barrier for commercial adoption.
• While it supports many languages, the quality assessment for non-English languages relies on machine translation.