This repository provides a reference implementation for Text-to-LoRA (T2L), a method for adapting large language models (LLMs) to specific tasks using only textual descriptions. It targets researchers and practitioners seeking efficient LLM customization without extensive fine-tuning.

How It Works

T2L generates LoRA (Low-Rank Adaptation) adapters for LLMs based on natural language task descriptions. This approach leverages hypernetworks to predict adapter weights, enabling rapid adaptation by conditioning on task semantics rather than requiring direct task data for adaptation. This offers a more efficient and flexible way to specialize LLMs.

Quick Start & Requirements

• Install dependencies using uv and pip, including a specific flash-attention wheel:

  git clone https://github.com/SakanaAI/text-to-lora.git
  cd text-to-lora
  # Install uv if not present
  uv venv --python 3.10 --seed
  uv sync
  uv pip install https://github.com/Dao-AILab/flash-attention/releases/download/v2.6.3/flash_attn-2.6.3+cu123torch2.3cxx11abiFALSE-cp310-cp310-linux_x86_64.whl
  uv pip install src/fishfarm
  

• Requires a GPU with >16GB VRAM for demos.
• Login to Hugging Face CLI: uv run huggingface-cli login
• Download checkpoints: uv run huggingface-cli download SakanaAI/text-to-lora --local-dir . --include "trained_t2l/*"
• Demo UI: uv run python webui/app.py
• LoRA generation: uv run python scripts/generate_lora.py {T2l_DIRECTORY} {TASK_DESCRIPTION}
• Evaluation: uv run python scripts/run_eval.py --model-dir {base_model_dir} --lora-dirs {lora_dirs} --save-results --tasks {tasks}
• Full documentation and paper available via links in the README.

Highlighted Details

• Achieves competitive performance across various benchmarks (e.g., GSM8K, ArcC) compared to baseline LoRA methods.
• Demonstrates that even SFT-trained T2L can generate reasonable LoRAs from random descriptions.
• Supports multiple base models including Mistral-7B, Llama-3.1-8B, and Gemma-2-2b-it.
• Offers both SFT and reconstruction training pipelines for T2L models.

Maintenance & Community

• The project is associated with Rujikorn Charakorn, Edoardo Cetin, Yujin Tang, and Robert Tjarko Lange.
• BibTeX entry provided for citation.

Licensing & Compatibility

• The repository does not explicitly state a license in the README. Users should verify licensing for commercial or closed-source use.

Limitations & Caveats

• Reproducibility notes mention small variances in evaluation results due to non-deterministic behavior in vLLM with LoRA.
• SFT training may require retrying due to potential Hugging Face datasets server connection issues when downloading many datasets.