Auto-CoT automates the design of Chain-of-Thought (CoT) prompts for large language models, reducing manual effort and matching or exceeding human-designed prompts. It targets researchers and practitioners working with LLMs who need to improve reasoning capabilities without extensive prompt engineering.

How It Works

Auto-CoT employs a diverse, multi-step prompting strategy. It generates multiple reasoning paths for a given problem and selects the most effective ones, enhancing robustness and performance compared to single-path CoT prompting. This approach aims to improve LLM reasoning by providing a more comprehensive and varied set of intermediate steps.

Quick Start & Requirements

• Install: pip install -r requirements.txt
• Prerequisites: Python >= 3.8, PyTorch 1.8.2+cu111, Torchtext 0.9.2. Datasets must be downloaded separately from provided GitHub links.
• Demo: python run_demo.py --task multiarith --pred_file log/multiarith_zero_shot_cot.log --demo_save_dir demos/multiarith
• Inference: python run_inference.py --dataset multiarith --demo_path demos/multiarith --output_dir experiment/multiarith
• More info: try_cot.ipynb

Highlighted Details

• Official implementation for the ICLR 2023 paper "Automatic Chain of Thought Prompting in Large Language Models."
• Matches or exceeds performance of manually designed CoT prompts on GPT-3.
• Focuses on saving manual effort in prompt design through automation.

Maintenance & Community

• Developed by Amazon Science.
• Paper: The Eleventh International Conference on Learning Representations (ICLR 2023)

Licensing & Compatibility

• License: Apache-2.0.
• Compatible with commercial use and closed-source linking.

Limitations & Caveats

The project requires specific, older versions of PyTorch (1.8.2+cu111) and Torchtext (0.9.2), which may pose compatibility challenges with newer environments. Datasets are not included and require manual download.