Meta-Prompting introduces a task-agnostic scaffolding technique to enhance Large Language Model (LLM) performance by orchestrating multiple instances of the same LLM as specialized "experts." This approach aims to improve accuracy and robustness across diverse tasks, simplifying user interaction by eliminating the need for task-specific instructions.

How It Works

Meta-prompting transforms a single LLM into a conductor that deconstructs complex tasks into subtasks. Each subtask is then assigned to a distinct "expert" instance of the LLM, guided by tailored instructions. The conductor LLM manages communication, integrates expert outputs, and performs verification, effectively acting as both an orchestrator and a panel of experts. This method is designed to be zero-shot and task-agnostic, simplifying usage.

Quick Start & Requirements

• Install dependencies: pip install -r requirements.txt
• Set OpenAI API keys as environment variables (e.g., export OPENAI_API_KEY="YOUR_API_KEY").
• Example execution: python run_experiments.py --task_name "GameOf24" --meta_config_path "prompts/meta-v0-2023-08-14-baseline.json" --model_name "gpt-3.5-turbo"
• Evaluation: python evaluate_outputs.py --directory "outputs/*/" --task "GameOf24"
• Data and prompts are available in /data and /prompts directories, respectively.
• Official implementation: utils/meta_scaffolding.py
• Datasets are available on Hugging Face: https://huggingface.co/datasets/turingmachine/meta-prompting

Highlighted Details

• Achieves significant accuracy improvements over standard, zero-shot chain-of-thought, expert, and multipersona prompting.
• Demonstrates a 17.1% average improvement over standard prompting when augmented with a Python interpreter.
• Integrates seamlessly with external tools like a Python interpreter.
• Tested extensively with GPT-4 and GPT-3.5-turbo.

Maintenance & Community

• Developed by Mirac Suzgun and Adam Tauman Kalai, affiliated with Microsoft Research New England.
• Work completed in Summer 2023.
• No explicit community links (Discord/Slack) or roadmap are provided in the README.

Licensing & Compatibility

• The README does not explicitly state a license.
• Code is provided for research and adaptation.

Limitations & Caveats

• The project is presented as a research study, and its stability for production use is not guaranteed.
• Specific performance claims are based on experiments conducted in August 2023.
• The README does not detail compatibility with models other than OpenAI's GPT series.