Summary

Terminal-Bench 3.0 is a benchmark designed to measure the capabilities of AI agents and language models in performing complex, realistic tasks within containerized command-line environments. It serves as a critical evaluation tool for frontier AI labs, aiming to push the boundaries of current model performance by introducing more challenging and valuable tasks than previous versions.

How It Works

The framework provides a suite of realistic tasks executed within isolated, containerized environments. Agents interact with a command-line interface to achieve objectives, simulating real-world computational work. This approach allows for standardized, reproducible evaluation of agent performance on complex problem-solving, distinguishing it by its focus on practical utility and challenging task design that aims to surpass current model limitations.

Quick Start & Requirements

• Installation: Install the harbor tool via uv tool install harbor.
• Running Benchmark: Execute tasks using uv run harbor run --repo harbor-framework/terminal-bench-3 --agent <agent_name> --env modal. For specific agent/model testing, include --model <model_name>.
• Prerequisites: Requires the uv package manager and the harbor tool. The --env modal flag suggests integration with Modal for execution environments.
• Resources: Supports high concurrency (--n-concurrent 100). Specific hardware requirements are not detailed, but execution implies computational resources for agent operation and task environments.
• Documentation: Harbor Documentation is available.

Highlighted Details

• Widely adopted by "virtually all frontier labs" for evaluating AI agent capabilities.
• Actively under development as the next generation of the Terminal Bench benchmark.
• Focuses on creating "challenging set of realistic and valuable tasks" to test advanced models.
• Features an automated PR review process for task contributions, ensuring high quality.
• Extensive list of authors indicates significant collaborative effort.

Maintenance & Community

The project is hosted by Stanford University and the Laude Institute, with a very extensive list of contributors. Community discussion and feedback, particularly for task proposals, are encouraged via Discord channels (#tb-task-proposals, #tb-3). The project is presented as an "ongoing effort," suggesting active development.

Licensing & Compatibility

The provided README does not specify a software license. This omission requires clarification regarding terms of use, distribution, and commercial compatibility.

Limitations & Caveats

Terminal-Bench 3.0 is explicitly described as a "work-in-progress preview." Users are encouraged to report bugs and suggest improvements, indicating that the benchmark may be incomplete or contain issues. Specific unsupported platforms or known bugs are not detailed in the README.