HackSynth provides an LLM-based agent and evaluation framework for autonomous penetration testing. It is designed for researchers and engineers seeking to benchmark and develop LLM agents for cybersecurity tasks, offering a standardized approach to evaluating their capabilities in simulated environments.

How It Works

HackSynth employs a dual-module architecture comprising a Planner and a Summarizer. The Planner module is responsible for generating sequences of commands to execute within a target environment, while the Summarizer processes the feedback received from these commands. This iterative loop allows the agent to autonomously navigate, identify vulnerabilities, and achieve penetration testing objectives.

Quick Start & Requirements

To set up HackSynth, first create accounts on Hugging Face and Neptune.ai, then copy your API keys into a .env file. Install the environment using:

python -m venv cyber_venv
source cyber_venv/bin/activate
pip install -r requirements.txt

Ensure desired CUDA devices are configured. The project includes setup instructions for PicoCTF and OverTheWire benchmarks. To start the agent and benchmark, run:

python run_bench.py -b benchmark.json -c config.json

The paper detailing HackSynth is available on arXiv: https://arxiv.org/abs/2412.01778.

Highlighted Details

• Features a novel LLM-based agent designed for autonomous penetration testing.
• Introduces two new CTF-based benchmark sets utilizing PicoCTF and OverTheWire platforms.
• Benchmarks comprise two hundred challenges across diverse domains and difficulties, offering a standardized evaluation framework.

Maintenance & Community

The project is contributed by Lajos Muzsai, David Imolai, and András Lukács. A related project, HackSynth-GRPO, focuses on reinforcement learning for cryptographic CTFs.

Licensing & Compatibility

HackSynth is released under the GNU AGPLv3 license. This strong copyleft license requires that any derivative works or modifications distributed must also be made available under the AGPLv3. This may impose restrictions on integration with closed-source commercial software.

Limitations & Caveats

The provided README does not explicitly detail any limitations, alpha status, or known bugs of the HackSynth framework.