This repository provides a framework for automated prompt engineering, enabling users to discover optimal prompts for large language models (LLMs) like GPT and Claude. It's designed for researchers, developers, and anyone seeking to maximize LLM performance through systematic experimentation and evaluation.

How It Works

The core approach involves generating a diverse set of candidate prompts based on a user-defined use case and test cases. These prompts are then systematically tested against the provided examples. An ELO rating system is employed to rank the prompts based on their performance, allowing users to identify the most effective ones. Specialized notebooks cater to classification tasks and offer advanced features like auto-generating test cases and optimizing Claude 3 Opus for cost and latency via Haiku.

Quick Start & Requirements

• Run via provided Jupyter notebooks (.ipynb).
• Requires Python 3.x.
• OpenAI API key for GPT models.
• Anthropic API key for Claude models.
• Optional: Weights & Biases for logging, Portkey for tracing.
• Official notebooks: gpt-prompt-engineer.ipynb, claude-prompt-engineer.ipynb, opus-to-haiku-conversion.ipynb

Highlighted Details

• Automated prompt generation, testing, and ELO-based ranking.
• Support for GPT-4, GPT-3.5-Turbo, and Anthropic's Claude 3 (Opus, Haiku).
• Claude 3 Opus -> Haiku conversion for cost and latency optimization.
• Classification-specific notebook for evaluating prompt correctness.
• Optional integration with Weights & Biases and Portkey for enhanced tracking.

Maintenance & Community

• Project maintained by Matt Shumer (@mattshumer_).
• Contributions are welcomed.
• Link to project: https://github.com/mshumer/gpt-prompt-engineer

Licensing & Compatibility

• MIT License.
• Permissive for commercial use and integration into closed-source projects.

Limitations & Caveats

The effectiveness of generated prompts is highly dependent on the quality and comprehensiveness of the user-provided use case and test cases. The Claude 3 Opus -> Haiku conversion notebook is experimental and may require fine-tuning for specific use cases.