This repository demonstrates the experimental use of OpenAI's GPT-3 (text-davinci-003) for identifying security vulnerabilities in code. It targets developers and security professionals interested in AI-assisted code analysis, showcasing GPT-3's potential to find a significant number of vulnerabilities, often exceeding commercial tools in quantity, with a low false positive rate.

How It Works

The approach involves feeding individual code files to GPT-3, which has a limited context window. To overcome this, each file is scanned separately. GPT-3 leverages its pre-existing knowledge of common libraries (like Express.js, Flask, C standard library) to infer vulnerabilities even without direct access to the library source code. This method is advantageous as it mimics how some static analysis tools operate and capitalizes on the LLM's broad training data.

Quick Start & Requirements

• Install/Run: The repository primarily serves as a demonstration; direct execution commands for a scanner are not provided. The core functionality relies on interacting with the OpenAI API.
• Prerequisites: Access to OpenAI's GPT-3 API (text-davinci-003) is required.
• Resources: Interaction with the API incurs costs. Analysis of the provided vulnerable code snippets is the primary activity.
• Links: Medium Article

Highlighted Details

• GPT-3 detected 213 vulnerabilities across 129 files, compared to 99 by a commercial tool (Snyk Code).
• Manual review of 60 GPT-3 findings yielded only 4 false positives (a 93.3% accuracy rate on the sampled findings).
• The tool successfully identified vulnerabilities in various languages including C, C#, Java, JavaScript, PHP, Python, Ruby, Go, and others.
• The experiment highlights the potential for "shift-left" security by integrating AI into the early stages of development.

Maintenance & Community

• The project appears to be a personal experiment by Chris Koch.
• No specific community channels (Discord, Slack) or active development roadmap are indicated.

Licensing & Compatibility

• The repository itself does not specify a license. The code snippets within are sourced from snoopysecurity/Vulnerable-Code-Snippets, which may have its own licensing.

Limitations & Caveats

GPT-3's inability to process entire repositories at once may limit its effectiveness in detecting vulnerabilities that span multiple files or require deep inter-file context. The experiment acknowledges that GPT-3 missed some vulnerabilities that an experienced human auditor would find.