Terminal-Bench provides a benchmark for evaluating AI agents on complex, real-world terminal tasks. It targets developers building LLM agents, benchmarking frameworks, or testing system-level reasoning, offering a reproducible suite for practical, end-to-end performance assessment.

How It Works

The project comprises a dataset of tasks and an execution harness. Each task includes an English instruction, a verification script, and an oracle solution. The harness connects language models to a sandboxed terminal environment, enabling autonomous execution and evaluation of agent capabilities in text-based interactions.

Quick Start & Requirements

• Install via pip: pip install terminal-bench or uv tool install terminal-bench.
• Prerequisites: uv, Docker.
• Documentation: Task Gallery, Dashboard Documentation.

Highlighted Details

• Beta release with ~100 tasks, with plans for expansion.
• Leaderboard available for submitting agent evaluations.
• Supports custom task creation and contribution.
• Includes bibtex citation for academic use.

Maintenance & Community

• Open to community contributions, especially for new tasks.
• Leaderboard Submission Guide.

Licensing & Compatibility

• License: Apache 2.0.
• Compatible with commercial and closed-source applications.

Limitations & Caveats

The project is currently in beta, with ongoing expansion of its task dataset.