ToolBench is an open platform for training, serving, and evaluating Large Language Models (LLMs) for tool learning. It provides a comprehensive dataset of over 16,000 real-world APIs and a framework for fine-tuning LLMs to master these APIs, enabling them to perform complex tasks involving single and multi-tool interactions. The project targets researchers and developers looking to enhance LLM capabilities in practical API usage.

How It Works

ToolBench leverages a large-scale dataset automatically generated using ChatGPT with enhanced function call capabilities. It employs a novel Depth-First Search based Decision Tree (DFSDT) method for data annotation, which improves efficiency and handles complex instructions requiring planning and reasoning. The platform includes an API retrieval mechanism to equip LLMs with open-domain tool-using abilities and offers a robust evaluation suite, ToolEval, which correlates highly with human judgment.

Quick Start & Requirements

• Install: pip install -r requirements.txt (Python >= 3.9)
• Data: Download data.zip from Google Drive.
• Dependencies: Python 3.9+, potentially CUDA for fine-tuning.
• Resources: Fine-tuning requires significant GPU resources (e.g., 2 x A100 80GB for ToolLLaMA-7b).
• Links: ToolBench GitHub, Web Demo, Paper

Highlighted Details

• Dataset: 3451 tools, 16464 APIs, 126K+ instances, 469K+ API calls.
• Models: Releases ToolLLaMA (v1, v2) fine-tuned on LLaMA-2, achieving performance comparable to ChatGPT in tool use.
• Evaluation: ToolEval provides reliable pass rate and win rate metrics, validated against human evaluation.
• RapidAPI Integration: Offers a backend service to use RapidAPI without personal keys.

Maintenance & Community

The project is actively maintained with regular updates, including new versions of ToolEval and models like ToolLLaMA-2. A Discord server is available for community engagement.

Licensing & Compatibility

Distributed under the Apache License 2.0. Intended solely for research and educational purposes.

Limitations & Caveats

Customized API usage is currently limited to closed-domain settings, with open-domain support planned. The project requires substantial computational resources for fine-tuning.