EdgeBench: Evaluating AI Agent Learning in Real-World Environments

EdgeBench is a comprehensive benchmark suite designed to evaluate the learning capabilities of AI agents within realistic, real-world environments. It moves beyond traditional one-shot performance metrics by enabling agents to learn and improve over extended periods, typically 12+ hours per task. This approach allows for the tracking of an agent's full learning trajectory, providing deeper insights into their scaling laws and developmental dynamics. The benchmark is particularly valuable for researchers and developers focused on creating advanced autonomous AI systems.

How It Works

The EdgeBench benchmark is powered by SForge, a robust two-container evaluation harness built for long-horizon agent assessment. Each task is encapsulated within isolated "work" and "judge" Docker containers, ensuring that agents only interact with the task environment and cannot exploit the evaluation process. Agents iteratively submit solutions and receive granular feedback, fostering a closed-loop learning process until the time limit is reached. Mechanisms like stop hooks and auto-resume support long-duration execution, while a Kubernetes backend enables large-scale parallel evaluations.

Quick Start & Requirements

• Primary Install: pip install sforge
• Prerequisites: Docker Engine running on a Linux host. Kubernetes is recommended for full-suite runs.
• Setup:
  1. Download task definitions: sforge fetch-tasks edgebench
  2. Pull pre-built Docker images: sforge pull --task <task_name> --registry seededge
  3. Start judge server (in a separate terminal): sforge serve
  4. Run an agent: SFORGE_AGENT_API_KEY="sk-xxx" sforge run --task <task_name> --agent <agent_name> --model "<model_name>" --timeout 43200 --run-id <run_id>
• Documentation: bytedance-seed.github.io/EdgeBench

Highlighted Details

• Comprises 134 real-world tasks, with 51 tasks and the evaluation framework publicly released.
• Evaluates agents over extended periods (12+ hours per task) to capture full learning improvement trajectories.
• Analysis reveals performance follows a log-sigmoid scaling law as a function of interaction time ($R^2 = 0.998$).
• Tasks are designed with high performance ceilings, reflecting significant human expert effort (averaging 57.2 hours per task).

Maintenance & Community

This project is developed by ByteDance Seed. For inquiries regarding evaluation on the full 134-task suite, contact zhongshu@bytedance.com.

Licensing & Compatibility

• EdgeBench Tasks: Released under CC BY 4.0, requiring attribution.
• SForge (evaluation harness): Released under the Apache License 2.0, which is generally permissive for commercial use and integration.

Limitations & Caveats

• Running the full 134-task benchmark is resource-intensive and costly, with a single 12-hour task potentially costing hundreds to over a thousand USD, and a full suite run estimated at a five-figure spend.
• Only a subset of 51 tasks is publicly available; access to the complete benchmark requires direct contact.
• The evaluation framework necessitates a Linux host with Docker Engine, and large-scale evaluations require a Kubernetes cluster.