EvoPrompt automates the creation of effective prompts for Large Language Models (LLMs) by integrating evolutionary algorithms (EAs) with LLM capabilities. This framework targets researchers and practitioners seeking to enhance LLM performance across diverse tasks without extensive manual prompt engineering, offering significant improvements over human-designed prompts and existing automated methods, thereby saving considerable development time and resources.

How It Works

EvoPrompt leverages evolutionary algorithms, specifically Genetic Algorithm (GA) and Differential Evolution (DE), to optimize discrete, human-readable prompts. The process begins with an initial population of prompts, which are then iteratively refined. LLMs are employed to generate new prompt candidates based on evolutionary operators like mutation and crossover. These candidates are evaluated on a development set, and the population is updated based on their performance, favoring better-performing prompts. This method uniquely combines the powerful language processing capabilities of LLMs with the efficient, gradient-free optimization strengths of EAs, enabling robust prompt discovery.

Quick Start & Requirements

• Installation: pip install -r requirements.txt
• Data: Download language understanding datasets to ./data/cls/{dataset_name}. Download BBH datasets from the CoT-hub repository to BBH/data/{dataset_name}.
• Prerequisites: An OpenAI API key is required and should be configured in auth.yaml. The framework supports evolution using text-davinci-003, gpt-3.5-turbo, and gpt-4.
• Links: BBH data source: CoT-hub (specific URL not provided). Example run scripts are available in the scripts/ directory for various tasks (classification, simplification, summarization, BBH).

Highlighted Details

• Demonstrates significant performance gains, achieving up to a 25% improvement on BIG-Bench Hard (BBH) tasks compared to human-engineered prompts and prior automated methods.
• Successfully optimizes prompts for both closed-source (e.g., GPT-3.5) and open-source LLMs.
• Validated effectiveness across a range of tasks including language understanding, text generation, and challenging BBH benchmarks.
• Highlights the synergistic benefits derived from combining LLMs with conventional evolutionary computation techniques.

Maintenance & Community

The project acknowledges contributions from the authors of the associated ICLR'2024 paper. It adheres to the Microsoft Open Source Code of Conduct and requires contributors to agree to a Contributor License Agreement (CLA). The README does not provide specific community channels (e.g., Discord, Slack) or links to a roadmap.

Licensing & Compatibility

The specific open-source license governing this repository is not explicitly stated within the provided README. This omission represents a significant adoption blocker, as license clarity is crucial for determining usage rights, especially for commercial applications. No compatibility notes regarding commercial use or linking with closed-source projects are mentioned.

Limitations & Caveats

For certain tasks, evaluation metrics such as accuracy or F1 scores can be coarse-grained, potentially leading to suboptimal prompt selection during the evolutionary process. The README indicates that the exploration of more complex tasks remains an area for future development. The absence of a clearly defined license is a critical limitation impacting the assessment of adoption viability and legal compliance.