Llama-X is an open academic research project aiming to systematically improve the LLaMA model to state-of-the-art performance through community collaboration. It targets researchers and developers interested in advancing LLM capabilities by providing transparent access to code, data, model weights, and experimental details, fostering rapid, iterative improvements.

How It Works

The project follows a versioned improvement strategy, with each new model release expected to demonstrate significant performance gains over its predecessor on a defined set of benchmarks. It focuses on ten key research areas, including instruction tuning, RLHF, data quality, long context, multi-modal, multilingual capabilities, infrastructure efficiency, evaluation, interpretability, and LLM actions. The approach emphasizes full fine-tuning rather than parameter-efficient methods like LoRA.

Quick Start & Requirements

• Install: Conda environment setup with PyTorch 1.12.0, Transformers 4.31.0, and CUDA 11.3.
• Prerequisites: Python 3.10, Conda, DeepSpeed, Hugging Face Transformers, and LLaMA model weights.
• Setup: Requires downloading LLaMA weights and converting them to Hugging Face format. Training commands are provided for single and multi-node setups using DeepSpeed Zero-3.
• Demo: Available via a demo page.

Highlighted Details

• Claims to achieve comparable performance to gpt-35-turbo with version 3.5.0 and is closing the gap with GPT-4.
• Reports efficient training: 7B model fine-tuned in 1 hour on 8x V100 GPUs with a batch size of 64.
• Focuses on full fine-tuning, not LoRA.
• Comprehensive evaluation plan includes Type-A (MMLU, HumanEval, GSM-8K, etc.) and Type-B benchmarks.

Maintenance & Community

• Actively seeking contributors for code, data, papers, and computing resources.
• Communication channels include GitHub Issues and email (llama-x@mail.com).
• Inspired by Meta AI LLaMA, Huggingface Transformers, and Llama Alpaca.

Licensing & Compatibility

• Resources are limited to academic research and prohibited for commercial purposes.
• No explicit license is stated, but the commercial use restriction is a significant caveat.

Limitations & Caveats

The project explicitly states that the use of its resources is limited to academic research and prohibits commercial use. The accuracy of model output is not guaranteed due to inherent LLM randomness.