Summary

CVE-Bench provides a standardized benchmark for evaluating AI agents' capabilities in exploiting real-world web application vulnerabilities. It addresses the need for robust security assessment of AI systems by offering a diverse set of critical CVE tasks. This resource is invaluable for AI safety researchers and developers aiming to build more secure and reliable AI agents.

How It Works

The benchmark comprises 40 critical-severity Common Vulnerability and Exposures (CVEs) sourced from the National Vulnerability Database. For each CVE, AI agents are tasked with executing attacks against target web applications to achieve specific outcomes, including Denial of Service, Remote Code Execution, unauthorized administrator login, and more. Evaluations are conducted using the Inspect framework, ensuring reproducibility and security isolation through Docker containers.

Quick Start & Requirements

• Installation: Requires Docker for reproducible evaluations and uv for dependency management (uv sync --dev).
• Prerequisites: Docker, uv.
• Hardware: Recommended for amd64 machines; arm64 support is experimental.
• Documentation: Links to the Inspect framework evaluation reference (https://inspect.aisi.org.uk/reference/inspect_eval.html) are provided.

Highlighted Details

• Features 40 critical-severity CVE tasks covering 8 distinct exploit outcome categories.
• Accepted as a spotlight paper at ICML 2025 and recognized with awards at Berkeley RDI's AgentX Competition and SafeBench.
• Employs Docker for isolated and reproducible evaluation environments.
• Utilizes the Inspect framework for standardized evaluation execution.

Maintenance & Community

The US AI Safety Institute contributed to the benchmark's development. No specific community channels (e.g., Discord, Slack) or detailed roadmap are mentioned in the provided text.

Licensing & Compatibility

The license type is not explicitly stated in the provided README, which may require further investigation for commercial or closed-source integration.

Limitations & Caveats

Experimental support exists for arm64 architectures. To prevent data contamination, not all manual exploits are released, though open-sourced graders are available for evaluation purposes.