OpenAlpaca provides fully open-source, instruction-following large language models based on OpenLLaMA. It targets researchers and developers seeking a permissively licensed alternative to proprietary models, enabling academic and commercial use with a relatively fast fine-tuning process.

How It Works

OpenAlpaca fine-tunes the OpenLLaMA base model using a dataset derived from databricks-dolly-15k, filtered for length. The fine-tuning process employs specific prompt formats to guide the model in understanding and responding to instructions, with or without additional context. The project emphasizes its Apache 2.0 license for the model weights and CC BY-SA 3.0 for the data.

Quick Start & Requirements

• Install dependencies: pip install -r requirements.txt
• PyTorch installation with CUDA 11.7 is recommended: pip install torch==1.13.1+cu117 -f https://download.pytorch.org/whl/torch/
• Fine-tuning requires 8xA100 GPUs and CUDA 11.7.
• Model weights can be loaded using Hugging Face Transformers.
• See Model Weights for specific model details.

Highlighted Details

• Fine-tuning completed in ~30 minutes on 8xA100 GPUs.
• Offers models based on OpenLLaMA-3B and OpenLLaMA-7B.
• Data preprocessing script process_dataset.py is available.
• Includes scripts for converting DeepSpeed checkpoints to PyTorch format.

Maintenance & Community

• Project last updated May 27, 2023.
• Relies on OpenLLaMA checkpoints for updates.
• Aims for rigorous evaluation against other models.

Licensing & Compatibility

• Model weights: Apache 2.0 license (permissive for commercial use).
• Training data: CC BY-SA 3.0 license (permissive for commercial use).

Limitations & Caveats

The models are fine-tuned on "previewed" versions of OpenLLaMA, suggesting potential for improvement with newer base model releases. The project plans future rigorous evaluations.