Summary

BoxPwnr is a modular framework for benchmarking LLMs and agentic strategies on cybersecurity challenges across platforms like HackTheBox and TryHackMe. It enables LLMs to autonomously solve security problems, providing detailed traces for analysis and a standardized method to evaluate AI performance in security contexts. The project aims to push AI capabilities in complex, multi-step problem-solving.

How It Works

The framework runs LLMs within a Dockerized Kali Linux environment, automating challenge-solving. An iterative loop is central: the LLM receives a system prompt, suggests a command, executes it in the Docker container, and analyzes the output. This cycle repeats until a flag is found or limits are hit. BoxPwnr mandates fully automated command execution, requiring LLMs to script interactions, handle service delays, and implement timeouts.

Quick Start & Requirements

Installation involves cloning with submodules (git clone --recurse-submodules), setting up the Python environment with uv (curl -LsSf https://astral.sh/uv/install.sh | sh then uv sync), and ensuring Docker is running. The Docker container builds on first run (~10 minutes). API keys (OpenAI, Anthropic, DeepSeek) are required on initial execution.

Highlighted Details

• Broad Platform Support: Integrates with HackTheBox (machines, CTFs, challenges), TryHackMe, PortSwigger Labs, Cybench, picoCTF, CTFd, and more.
• Flexible Solver Architectures: Offers chat, chat_tools, claude_code, hacksynth, and external solver options for varied agentic strategies.
• Comprehensive Traces & Benchmarks: Generates detailed solving traces with logs, command outputs, and reasoning, viewable in an interactive web viewer. Extensive benchmark data quantifies solved challenges.
• Automated Execution: Designed for LLMs to provide fully automated commands, scripting interactions and managing timeouts.

Maintenance & Community

No specific details on maintainers, community channels, or active development signals were present in the provided README content.

Licensing & Compatibility

The README content does not explicitly state the project's license or provide compatibility notes for commercial use.

Limitations & Caveats

Presented as a "fun experiment" and research tool, it may not be production-ready. Effectiveness depends on the LLM's capabilities and challenge complexity. Reliance on external API keys introduces potential costs.