Summary

ExploitBench benchmarks AI agents' ability to perform complex software exploitation tasks, from identifying vulnerabilities to achieving arbitrary code execution. It targets AI researchers and model providers, offering a standardized ladder for evaluating and comparing agent capabilities through reproducible experiments.

How It Works

The system drives AI models via direct APIs or OpenAI-compatible gateways, executing them within containerized environments featuring an MCP server. It measures performance across a multi-stage exploitation ladder, specifically detailing capabilities within the Chromium V8 environment using pre-built Docker images. Configuration is managed via YAML files, allowing flexible definition of models, environments, and experimental parameters.

Quick Start & Requirements

• Install: make install, source .venv/bin/activate.
• Configure: Set LLM API keys in .env, run exploitbench doctor for verification.
• Prerequisites: Python virtual environment, Docker, LLM API keys.
• Run: make smoke (basic test), exploitbench benchmark --config benchmarks/v8.yaml ... (full runs).
• Docs/Results: exploitbench.ai, docs/architecture.md, docs/RUNBOOK.md.

Highlighted Details

• Evaluates AI agents across a detailed exploitation ladder for V8 vulnerabilities.
• Supports diverse LLM providers (Anthropic, OpenAI, Gemini, etc.) via native SDKs or LiteLLM.
• Leverages pre-built, large Docker images (~70 GB per bug) for V8 environments, simplifying setup.
• Provides a public leaderboard, capability heatmaps, and per-CVE drilldowns at exploitbench.ai.
• Enables controlled experiments with cost (--cost-cap-usd) and turn (--turn-budget) limits.

Maintenance & Community

Direct support is offered to academic researchers and model providers via contact@exploitbench.ai. Users are advised against performing reinforcement learning on the benchmark.

Licensing & Compatibility

The project's license is not specified in the README, necessitating clarification for adoption decisions, particularly regarding commercial use or derivative works.

Limitations & Caveats

Reinforcement learning experiments are discouraged. The absence of a stated license is a significant adoption blocker. Setup requires managing large Docker images and LLM API credentials.