promptomatix  by SalesforceAIResearch

LLM prompt optimization framework

Created 6 months ago
857 stars

Top 41.8% on SourcePulse

GitHubView on GitHub
Project Summary

<2-3 sentences summarising what the project addresses and solves, the target audience, and the benefit.> Promptomatix is an AI-driven framework automating and optimizing prompts for Large Language Models (LLMs). It tackles manual prompt engineering's inefficiencies with a structured, iterative approach using DSPy and advanced optimization. This benefits researchers and developers by ensuring prompt consistency, cost-effectiveness, and high-quality LLM outputs, reducing trial-and-error.

How It Works

The architecture includes Input Processing, Synthetic Data Generation, an Optimization Engine (DSPy/meta-prompt backends), and an Evaluation System with Feedback Integration. It analyzes input, creates tailored synthetic datasets, iteratively refines prompts based on task requirements and metrics, and incorporates human feedback for continuous improvement, aiming for efficient prompt discovery.

Quick Start & Requirements

  • Installation: Clone repo, cd into it, run ./install.sh. Sets up Python 3 virtual environment (promptomatix_env), initializes submodules (DSPy), and installs dependencies.
  • Activation: source promptomatix_env/bin/activate before use.
  • API Keys: Configure LLM provider keys via environment variables (e.g., OPENAI_API_KEY) or .env file.
  • Prerequisites: Python 3. Requires API keys for supported LLM providers (OpenAI, Anthropic, Cohere).
  • Testing: python -m src.promptomatix.main --raw_input "Given a questions about human anatomy answer it in two words" --model_name "gpt-3.5-turbo" --backend "simple_meta_prompt" --synthetic_data_size 10 --model_provider "openai".
  • Docs: API docs in docs/; examples in examples/notebooks/.

Highlighted Details

  • Zero-Configuration Intelligence: Automates task analysis, technique selection, and prompt configuration.
  • Automated Dataset Generation: Creates domain-specific synthetic training and testing data.
  • Task-Specific Optimization: Dynamically selects appropriate DSPy modules and metrics.
  • Framework Agnostic: Supports multiple LLM providers (OpenAI, Anthropic, Cohere).
  • Flexible Interfaces: Offers CLI and REST API access.

Maintenance & Community

Standard contribution guidelines and code of conduct are provided. Contact information for Rithesh Murthy is available for inquiries and contributions. No specific community channels or details on major contributors/sponsorships were found.

Licensing & Compatibility

A LICENSE.txt file is present for licensing information. Specific license type, commercial use, or closed-source linking compatibility details are not explicitly stated.

Limitations & Caveats

The provided text does not explicitly list limitations, unsupported platforms, known bugs, or alpha status. Functionality depends on configured LLM provider API access.

Health Check
Last Commit

5 months ago

Responsiveness

Inactive

Pull Requests (30d)
1
Issues (30d)
0
Star History
725 stars in the last 30 days

Explore Similar Projects

Starred by Eric Zhu Eric Zhu(Coauthor of AutoGen; Research Scientist at Microsoft Research) and Chip Huyen Chip Huyen(Author of "AI Engineering", "Designing Machine Learning Systems").

PromptWizard by microsoft

0.2%
4k
Agent-driven framework for task-aware prompt optimization
Created 1 year ago
Updated 3 months ago
Feedback? Help us improve.