This repository provides a lightweight, opinionated codebase for memory-efficient fine-tuning of Mistral models using LoRA. It targets researchers and developers looking to adapt Mistral's models for specific tasks with a focus on ease of use and performance, particularly in multi-GPU, single-node setups.

How It Works

The codebase leverages LoRA (Low-Rank Adaptation), a parameter-efficient fine-tuning technique that freezes most pre-trained weights and trains only a small percentage of additional low-rank matrix perturbations. This approach significantly reduces memory requirements and computational cost compared to full fine-tuning, enabling efficient adaptation of large language models.

Quick Start & Requirements

• Install: pip install -r requirements.txt
• Prerequisites: Python, PyTorch. Recommended: A100 or H100 GPU for optimal performance.
• Data Format: Strict .jsonl format for pretraining ({"text": "..."}) or instruction tuning ({"messages": [...]}).
• Verification: Use ./utils/validate_data script to check data format and estimate training time.
• Official Docs: Mistral Finetune

Highlighted Details

• Supports Mistral Large v2 (123B) and Mistral Nemo (12B) models.
• Handles conversational and function-calling data formats.
• Includes utilities for data reformatting and model vocabulary extension.
• Integrates with Weights & Biases for experiment tracking.

Maintenance & Community

• Actively maintained by Mistral AI.
• Supports various Mistral model versions, including recent additions like Mistral Large v2 and Mistral Nemo.
• No explicit community links (Discord/Slack) are provided in the README.

Licensing & Compatibility

• The repository itself is not explicitly licensed in the README.
• It is designed to fine-tune Mistral models, which have their own licensing terms. Users must comply with the licenses of the base models they choose to fine-tune.

Limitations & Caveats

• Highly opinionated regarding data formatting, requiring custom reformatting scripts for non-standard datasets.
• Fine-tuning Mistral Large v2 and Mistral Nemo requires significantly more memory.
• MoE models may exhibit higher performance variance during fine-tuning, suggesting multiple runs for optimal results.