This project provides a production-ready SDK and CLI for optimizing Large Language Model (LLM) prompts using natural language feedback, addressing the limitations of traditional numerical scoring. It targets engineers and researchers seeking to enhance LLM performance through an interpretable and cost-effective feedback loop, enabling continuous prompt improvement post-deployment.

How It Works

The system employs a three-model loop: an Agent executes tasks with a given prompt, an Evaluator identifies failures and generates textual feedback, and an Optimizer revises the prompt based on this feedback. This approach leverages "English Error Terms"—natural language critiques—to guide prompt refinement, allowing LLMs to self-improve by adjusting instructions rather than model weights, leading to more precise and understandable optimization.

Quick Start & Requirements

• Installation: Install via pip: pip install prompt-learning. For development, clone the repository and install from source.
• Environment Setup: Set API keys for your chosen provider (OPENAI_API_KEY or GOOGLE_API_KEY/GEMINI_API_KEY).
• Providers: Supports OpenAI (GPT-4, GPT-4-turbo, GPT-3.5-turbo) and Google AI (Gemini 2.5-flash, 2.5-pro, 2.5-flash-image).
• Core Commands: prompt-learn optimize for text-based prompt optimization and prompt-learn image for image generation prompt iteration.
• Dataset: Requires input variables, an LLM output column, and one or more feedback columns containing natural language critiques.
• Dependencies: Python, LLM provider API access.

Highlighted Details

• English Error Terms: Utilizes natural language critiques for more granular and effective prompt optimization compared to scalar feedback.
• Online Prompt Management: Designed as a continuous improvement system suitable for production environments.
• Cost Efficiency: Aims for low latency, delivering strong results within minutes and offering built-in budget management.
• Single-Loop Success: Claims significant prompt improvements can be achieved within a single optimization cycle.
• Image Generation: Includes a dedicated CLI command for iterating and testing image generation prompts using Google's models.
• Cost Tracking: Features verbose mode for real-time cost estimates and budget enforcement to prevent overspending.

Maintenance & Community

The project is developed by Arize AI, with contributions from Nouamane Benbrahim and Priyan Jindal. For research or SDK-specific inquiries, contact pjindal@arize.com. The repository includes a clear architectural breakdown and development setup instructions.

Licensing & Compatibility

This project is licensed under the Elastic License 2.0 (ELv2). This license may have restrictions on use, particularly concerning offering the software as a managed service.

Limitations & Caveats

The README does not explicitly detail limitations such as alpha status, known bugs, or unsupported platforms. The effectiveness of the optimization heavily relies on the quality and expressiveness of the provided natural language feedback. While cost management is built-in, users must still monitor API usage and associated costs.