PromptWizard is a framework for optimizing prompts for Large Language Models (LLMs) through a self-evolving, task-aware mechanism. It is designed for researchers and practitioners seeking to improve LLM performance by iteratively refining prompts and in-context examples based on LLM-generated feedback and synthetic data.

How It Works

PromptWizard employs a feedback-driven refinement loop where the LLM itself generates, critiques, and synthesizes new prompts and examples. This process optimizes both instructions and in-context learning examples in tandem, aiming for holistic task performance improvement. It supports generating diverse, task-aware synthetic examples and incorporates self-generated Chain-of-Thought (CoT) steps, combining positive, negative, and synthetic examples for robust prompt evolution.

Quick Start & Requirements

• Install via pip install -e . after cloning the repository and setting up a Python virtual environment.
• Requires API access to LLMs (OpenAI or Azure OpenAI). Configuration is managed via .env and promptopt_config.yaml.
• Supports datasets like GSM8k, SVAMP, AQUARAT, and Instruction_Induction (BBII). Custom datasets require a .jsonl format with question and answer fields.
• Experimentation suggests optimization takes 20-30 minutes on average for supported datasets.
• Detailed usage examples are available in the demos folder and a linked notebook.

Highlighted Details

• Achieves state-of-the-art performance across various tasks, outperforming other methods.
• Optimizes prompts and in-context examples concurrently for improved LLM task performance.
• Supports generating synthetic examples and reasoning chains (CoT) to enhance prompt robustness.
• Allows for task-specific data processing and evaluation logic.

Maintenance & Community

• Developed by Microsoft.
• Contributions are welcome, subject to a Contributor License Agreement (CLA).
• The project adheres to the Microsoft Open Source Code of Conduct.

Licensing & Compatibility

• The repository does not explicitly state a license in the provided README.

Limitations & Caveats

• Requires API access to LLMs, which incurs costs.
• Prompt optimization time is dataset-dependent and can be significant.
• Fine-tuning generated prompts may require user supervision for optimal task alignment.