Summary

Long-Horizon Terminal-Bench (LHTB) offers a 46-task benchmark for evaluating LLM agents in complex, stateful terminal environments over extended interactions. It targets researchers and engineers assessing agent capabilities, providing objective performance metrics via hidden, artifact-based verification, moving beyond self-reported progress.

How It Works

LHTB immerses LLM agents in a containerized terminal, simulating long-horizon tasks spanning hundreds of steps. Its core innovation is dense reward grading using hidden, rebuild-from-artifact verifiers for objective assessment. This prevents agents from gaming the system, focusing on demonstrable outcomes across diverse domains.

Quick Start & Requirements

• Install: harbor via uv tool install harbor or pip install harbor. Docker required.
• Prerequisites: Git LFS. Apple Silicon: export DOCKER_DEFAULT_PLATFORM=linux/amd64. API keys (e.g., OPENAI_API_KEY) as env vars.
• Setup: git clone ..., git lfs pull. Smoke test: harbor run -c configs/examples/oracle_smoke.yaml.
• Resources: Blog: https://zli12321.github.io/LHTB/. Leaderboard: https://zli12321.github.io/LHTB/leaderboard.html. Paper: https://arxiv.org/abs/2607.08964. Dataset: https://huggingface.co/datasets/IntelligenceLab/Long-Horizon-Terminal-Bench.

Highlighted Details

• Benchmark Hardness: Frontier models solve ~28% of tasks strictly; 29 tasks unsolved. ~55% of runs yield < 0.25 reward.
• Cost-Effectiveness: Reward vs. cost varies. Grok 4.5 ($11/task) leads, but Hy3 ($2.47/task) offers competitive value.
• Task Scope: 46 tasks: games, multimodal analysis, software/reverse engineering, scientific computing, earth/climate/energy, security, research reproduction, professional workflows.
• Evaluation Rigor: Hidden, artifact-based verifiers ensure objective performance via dense reward grading.

Maintenance & Community

Associated with Terminal-Bench/2.0 and Harbor harness. Resources: blog, leaderboard, paper, Hugging Face dataset. No direct community channels listed.

Licensing & Compatibility

• License: Apache License 2.0.
• Compatibility: Permissive, suitable for commercial use and closed-source integration.

Limitations & Caveats

Current LLM agents struggle with long-horizon, stateful terminal tasks, showing a significant capability gap. Full suite runs incur considerable API costs and computational resources. Some Docker images are amd64-specific, requiring platform configuration on architectures like Apple Silicon.