Code Alpaca is an open-source project providing a LLaMA-based language model fine-tuned for code generation tasks. It offers a dataset of 20,000 instruction-following examples specifically curated for code-related instructions, along with the code to replicate the fine-tuning process. This project is ideal for researchers and developers looking to build or experiment with instruction-tuned models for programming assistance.

How It Works

The project leverages the Self-Instruct methodology, adapting it for code-specific tasks. A dataset of 20,000 instruction-output pairs was generated using OpenAI's text-davinci-003, with modified prompts and seed tasks focused on code generation, editing, and optimization. This approach aims to create a cost-effective, specialized instruction-following dataset. The model is fine-tuned using Hugging Face's Transformers library and DeepSpeed, with specific hyperparameters detailed for reproducibility.

Quick Start & Requirements

• Install: pip install -r requirements.txt
• Prerequisites: OpenAI API key (for data generation), Hugging Face Transformers library (specific fork), DeepSpeed.
• Hardware: Recommended 8x A100 80GB GPUs; can run on 4x A100 with adjustments.
• Demo: https://code-alpaca-demo.vercel.app/

Highlighted Details

• Fine-tuned on 20K code-specific instruction-following examples.
• Data generation cost under $200.
• Utilizes a modified fork of Hugging Face Transformers for LLaMA compatibility.
• Training code includes DeepSpeed integration for distributed training.

Maintenance & Community

The project is primarily maintained by Sahil Chaudhary. Further community engagement details (e.g., Discord/Slack) are not specified in the README.

Licensing & Compatibility

The model weights are not released due to OpenAI TOS and LLaMA license restrictions. The code and dataset are available under a permissive license, but users must adhere to the LLaMA model's license for any derivative works or usage.

Limitations & Caveats

The model is not fine-tuned for safety or harmlessness. Model weights are not provided, requiring users to obtain and convert LLaMA checkpoints themselves. Evaluation results are pending.