Summary

ToolSandbox provides a stateful, conversational, and interactive benchmark for evaluating LLM tool-use capabilities, addressing limitations of existing stateless or single-turn assessments. It targets researchers and engineers, offering deeper insights into LLM performance on complex tasks involving state dependencies and implicit reasoning, revealing significant performance gaps.

How It Works

The system features a stateful execution context managing tools, dialog history, and world state. Tools are composable Python functions. It supports on-policy conversational evaluation via a user simulator and employs a Milestone Directed Acyclic Graph (DAG) for dynamic, flexible assessment of task completion over arbitrary trajectories, using similarity measures against predefined milestones.

Quick Start & Requirements

Installation requires a Python 3.9 virtual environment (e.g., Miniforge3) and pip install '.[dev]'. Essential prerequisites include API keys for various LLM providers (OpenAI, Anthropic, Google, Cohere) and services (RapidAPI), plus Google Cloud authentication for Gemini. Hosting open-source models may require tools like vLLM. Example commands illustrate scenario execution.

Highlighted Details

• Stateful Tool Execution: Models and evaluates implicit state dependencies between tools.
• Conversational Evaluation: Integrates a user simulator for on-policy dialogue testing.
• Dynamic Milestone Evaluation: Uses a Milestone DAG for flexible assessment of task completion.
• Challenging Scenarios: Includes State Dependency, Canonicalization, and Insufficient Information categories.
• Tool Augmentation: Supports techniques like distraction tools and scrambled names to increase difficulty.

Maintenance & Community

The project is associated with a research paper; no specific community channels or roadmap details are provided.

Licensing & Compatibility

The specific open-source license for ToolSandbox is not detailed in the provided README.

Limitations & Caveats

Tool code executes directly on the host, posing a potential risk. The system relies heavily on numerous external API keys. Setup complexity is moderate, and the project's license is unspecified.