A Snyk benchmark initiative
An attacker-reachable recall study of deterministic SAST and AI security agents
We ran nine productized security configurations once each, blind, on 20 open-source web applications. For the seven endpoint-aware configurations, a finding counts only when it locates the attacker-controlled source and the dangerous sink of a curated vulnerability, each within two lines; DeepSec is scored separately.
Headline evidence · Attacker-Reachable Vulnerability Recall
In this study neither engine was sufficient alone. A vulnerability class an engine does not report is a real miss here, because every reference vulnerability is attacker-reachable.
Dataset 2.0.0 · Snyk VulnBench 2.0 · 108 reference vulnerabilities · one run per configuration and application · Source: published 2.0 paper
| Configuration | Attacker-Reachable Vulnerability Recall (macro, matched) | 95% interval | Precision against the reference set (lower bound) | Matched at the exact line |
|---|---|---|---|---|
| Snyk Code | 82.4% (89 of 108) | 74.0–90.5% | 20.9% | 85 |
| Claude Code /security-review · Opus 5 xhigh | 79.3% (85 of 108) | 71.6–86.7% | 74.6% | 79 |
| Claude Code /security-review · Opus 5 medium + Snyk MCP | 75.2% (80 of 108) | 68.2–82.3% | 82.5% | 74 |
| Claude Code /security-review · Sonnet 5 xhigh | 67.0% (73 of 108) | 57.7–76.2% | 74.5% | 55 |
| Codex Security · GPT-5.6 Luna | 41.2% (45 of 108) | 26.6–56.5% | 20.8% | 15 |
| Codex Security · GPT-5.6 Terra | 30.1% (35 of 108) | 19.1–41.9% | 15.8% | 6 |
| Codex Security · GPT-5.6 Sol | 29.2% (34 of 108) | 18.1–41.1% | 12.7% | 2 |
An empty cell means not applicable or unavailable as stated in `columns`; it never means zero.
| Configuration | Localized recall (macro, matched) | 95% interval | Precision against the reference set (lower bound) |
|---|---|---|---|
| DeepSec · Opus 5 xhigh | 37.5% (37 of 108) | 22.6–53.2% | 7.1% |
| DeepSec · GPT-5.6 Sol xhigh | 2.5% (1 of 108) | 0.0–7.5% | 10.0% |
What the results mean in practice
Deterministic SAST and AI review covered different ground; what an AI engine reported depended on its harness as much as on its model; and coverage and cost were only loosely related.
Snyk Code matched 82.4% (89 of 108) and Claude Code /security-review with Opus 5 xhigh 79.3% (85 of 108). They disagree on 28 vulnerabilities, 16 matched only by Snyk Code and 12 only by Opus 5 xhigh; a corpus of this size cannot separate them.
Interpret with care: A union across configurations is a ceiling, not one tool's result. Combining the two also combines their queues: 539 findings for 101 matches.
Claude Code's /security-review excludes ReDoS and reports prototype pollution and open redirect only at extremely high confidence. On the 20 reference vulnerabilities in those classes, Snyk Code matched 18 and Opus 5 xhigh 9; on the 81 others (excluding 7 cross-site scripting vulnerabilities in React and Angular applications), Opus 5 xhigh matched 71 and Snyk Code 67.
Interpret with care: The policy reading rests on the policy text and one supplementary control run; harness and model cannot be fully separated by this design.
Snyk Code scanned all 20 applications in 8 minutes; the comparable AI configurations took between 2.4 hours and 8.9 hours.
Interpret with care: One run per configuration and application: every ratio is a single observation. Snyk Code has no per-scan model cost; its license cost is outside this study.
Snyk Code and Opus 5 xhigh are the two highest-ranked endpoint-aware configurations under every one of the 20 alternative scoring rules; Snyk Code ranks first under 18 and Opus 5 xhigh under the other two; their order should not be read as a result.
Interpret with care: Rescoring shows which conclusions survive changes in the scoring rule; it does not add statistical power.
Research principles
VulnBench is a versioned research initiative, not a universal leaderboard. Every release defines what it measured and what it did not prove.
Release history
Releases measure different things. JS 1.0 measured repeatability and agreement with Snyk Code; 2.0 measures attacker-reachable recall against an independent reference set.
View release catalogPublication
The paper, reviewed methodology, source snapshot, and release data use stable public links.
Preferred citation
@misc{tal2026snykvulnbench20,
author = {Tal, Liran and Liu, Chris and Fereday, Charlotte and Roth, Sebastian and Clark, Brian},
title = {How Deterministic SAST and AI Security Agents Divide the Work: Coverage, Harness Policy and Cost in Snyk VulnBench 2.0},
year = {2026},
eprint = {2610.38617},
archivePrefix = {arXiv},
primaryClass = {cs.CR},
doi = {10.48550/arXiv.2610.38617},
url = {https://arxiv.org/abs/2610.38617}
}@misc{tal2026snykvulnbenchjs10,
author = {Tal, Liran and Kloos, Johannes and Rudich, Arsenii and Thoemmes, Stephen and Nair, Manoj},
title = {Snyk VulnBench JS 1.0: Can LLMs Find the Same Bugs Twice?},
year = {2026},
url = {https://arxiv.org/abs/2606.15762}
}