LaMini-LM offers a diverse collection of small, efficient language models distilled from ChatGPT and trained on a 2.58M instruction dataset. Targeting researchers and developers seeking performant, compact LLMs, it provides a range of architectures and sizes for various NLP tasks.

How It Works

LaMini-LM employs offline distillation from GPT-3.5-turbo, generating 2.58M instruction-response pairs using prompts from existing resources like Self-Instruct, P3, Flan, and Alpaca. This approach allows for the creation of smaller, more manageable models that retain significant instruction-following capabilities, making them suitable for resource-constrained environments.

Quick Start & Requirements

• Install via pip: pip install -q transformers
• Models can be loaded using HuggingFace pipeline().
• Requires Python and the transformers library.
• See HuggingFace Hub for model checkpoints.

Highlighted Details

• Offers models based on T5, Flan-T5, Cerebras-GPT, GPT-2, and GPT-Neo architectures.
• Evaluated on 15 diverse NLP tasks using lm-evaluation-harness.
• Includes human evaluation results and qualitative analysis comparing LaMini-LM performance against Alpaca-7B.
• Models are available in various sizes, from 61M to 1.5B parameters.

Maintenance & Community

• The project is associated with mbzuai-nlp.
• Citation details are provided in BibTeX format.

Licensing & Compatibility

• Licensed under CC BY NC 4.0.
• Intended for research use only; commercial use is restricted.

Limitations & Caveats

The CC BY NC 4.0 license prohibits commercial use. The README notes that reported LLaMA results are not directly comparable due to insufficient detail for reproducible evaluation.