CodeGen is a family of open-source large language models for program synthesis, offering competitive performance against models like OpenAI Codex. It is designed for researchers and developers working on code generation, autocompletion, and other code-related AI tasks.

How It Works

CodeGen models are trained using the Jaxformer library, a framework for data pre-processing, training, and fine-tuning. The models leverage transformer architectures and are trained on a large corpus of code and natural language, enabling them to generate code snippets from natural language descriptions and perform multi-turn program synthesis.

Quick Start & Requirements

• Install/Run: Models are available via Hugging Face Hub. Usage involves importing AutoTokenizer and AutoModelForCausalLM from the transformers library.
• Prerequisites: Python, PyTorch, and the transformers library. Specific model versions may have varying hardware requirements (e.g., GPU recommended for larger models).
• Links:
  • Hugging Face Hub: https://huggingface.co/Salesforce
  • Jaxformer (Training): https://github.com/salesforce/jaxformer

Highlighted Details

• Family of models with sizes ranging from 350M to 16B parameters.
• CodeGen2.5 (7B) outperforms 16B models.
• CodeGen2.0 offers strong infill sampling capabilities.
• CodeGen1.0 was competitive with OpenAI Codex at its release.

Maintenance & Community

• Developed by Salesforce AI Research.
• Associated publications include ICLR 2023 papers for CodeGen and CodeGen2.

Licensing & Compatibility

• The README does not explicitly state a license. However, Salesforce AI models are typically released under research-focused licenses. Users should verify specific terms for commercial use or closed-source integration.

Limitations & Caveats

• The models are released for research purposes and may not be optimized for all downstream applications. Users are advised to evaluate and address potential concerns regarding accuracy, safety, and fairness before deployment, especially in high-risk scenarios.