Summary

PrimeVul provides a curated dataset for training and evaluating code language models in vulnerability detection. It features ~7,000 vulnerable and ~229,000 benign C/C++ functions across 140+ CWEs, emphasizing high-accuracy labeling and minimal contamination. This enables more realistic model assessment and development for practical security applications.

How It Works

This dataset reconstructs existing vulnerability corpora using novel, human-level accurate labeling techniques, reportedly three times better than prior automatic methods. Rigorous de-duplication and chronological splits minimize data contamination. Evaluation is enhanced by the "VD-Score" (Vulnerability Detection Score), which balances flaw detection against false alarms, and by paired samples that expose model weaknesses in identifying subtle vulnerabilities and patches. The v0.1 release adds commit, vulnerability (CVE, NVD), and file-level metadata for deeper contextual analysis.

Quick Start & Requirements

Installation uses Conda: conda env create -f environment.yml. Prerequisites include Python, Huggingface Accelerate (for models >7B), and specific transformers versions (e.g., 4.33.0 for CodeGen2.5). Accelerate configuration via accelerate config is necessary. Links to dataset releases (v0.1 and original) and experimental code are available within the repository.

Highlighted Details

• Dataset: ~7k vulnerable, ~229k benign C/C++ functions, 140+ CWEs.
• Labeling accuracy: Human-level, 3x better than automatic methods.
• Data integrity: Rigorous de-duplication and chronological splits.
• Evaluation: VD-Score for risk/false alarm balance; paired samples for model weakness analysis.
• v0.1: Enhanced with commit, vulnerability, and file-level metadata.

Maintenance & Community

Associated with authors of an ICSE 2025 accepted paper. Notably used to evaluate Gemini-1.5 for vulnerability detection. No community channels (e.g., Discord, Slack) or roadmap details are provided in the README.

Licensing & Compatibility

The specific open-source license for the PrimeVul dataset and associated code is not explicitly stated in the provided README content. Consequently, compatibility for commercial use or closed-source linking cannot be determined from this information.

Limitations & Caveats

The v0.1 release only includes samples for which metadata (CWE, CVE) could be successfully retrieved, meaning the full original dataset might contain more samples but with less metadata. Users should refer to the original release for the complete set used in the paper if metadata is critical.