VulnBench
Menu

Current release · 2.0

What do AI security agents find that SAST does not, and what do they miss that SAST catches?

A blind study of nine productized security configurations, from the Snyk Code static analyzer to Claude Code, Codex Security and DeepSec, on 108 hand-curated, attacker-reachable vulnerabilities in 20 open-source web applications.

Explore the resultsRead the paper
108
attacker-reachable vulnerabilities
20
open-source web applications
62
flows across more than one file
9
configurations, one run each

What the results mean in practice

An AI security engine is a model inside a harness.

Deterministic SAST and AI review covered different ground; what an AI engine reported depended on its harness as much as on its model; and coverage and cost were only loosely related.

  1. 01

    SAST and AI review cover different ground

    Snyk Code matched 82.4% (89 of 108) and Claude Code /security-review with Opus 5 xhigh 79.3% (85 of 108). They disagree on 28 vulnerabilities, 16 matched only by Snyk Code and 12 only by Opus 5 xhigh; a corpus of this size cannot separate them.

    Together they match 101 of 108, as many as all seven endpoint-aware configurations combined. The best pair without SAST, Opus 5 xhigh with Codex Security GPT-5.6 Terra, reaches 89 (93 under many-to-one credit).

    Snyk Code adds 16 vulnerabilities to Opus 5 xhigh, only 3 of them cross-file, led by open redirect (6). Opus 5 xhigh adds 12 to Snyk Code, 7 of them cross-file, including the authorization classes.

    Snyk Code is the only one of the seven endpoint-aware configurations with unique coverage: 10 vulnerabilities in six classes (5 under many-to-one credit).

    Outside the reference set, in this run, AI configurations found at least seven real vulnerabilities in scored classes that the reference set did not contain and Snyk Code did not report.

    Interpret with care: A union across configurations is a ceiling, not one tool's result. Combining the two also combines their queues: 539 findings for 101 matches.

    Open the evidence
  2. 02

    The harness decides what gets reported

    Claude Code's /security-review excludes ReDoS and reports prototype pollution and open redirect only at extremely high confidence. On the 20 reference vulnerabilities in those classes, Snyk Code matched 18 and Opus 5 xhigh 9; on the 81 others (excluding 7 cross-site scripting vulnerabilities in React and Angular applications), Opus 5 xhigh matched 71 and Snyk Code 67.

    In a supplementary run on 19 of the 20 applications, the same model at the same effort, without that policy, matched 16 of the 20.

    Codex Security with GPT-5.6 Sol located 80 vulnerabilities at file level but 34 under the two-line rule, typically pointing above the dangerous call.

    DeepSec's model-free scan stage queued a file from only 56 of the 108 flows before any model ran.

    The model still matters inside a harness: under the same harness, effort and prompt, Opus 5 matched 14 vulnerabilities that Sonnet 5 missed, against 2 the other way.

    Interpret with care: The policy reading rests on the policy text and one supplementary control run; harness and model cannot be fully separated by this design.

    Open the evidence
  3. 03

    Coverage and cost were only loosely related

    Snyk Code scanned all 20 applications in 8 minutes; the comparable AI configurations took between 2.4 hours and 8.9 hours.

    Within Codex Security, all at xhigh effort, GPT-5.6 Sol cost 35.0 times as much as GPT-5.6 Luna and matched 11 fewer vulnerabilities: a comparison of model tiers, not of reasoning effort. Within Claude Code /security-review, at the same xhigh effort and with the same prompt, Opus 5 cost 4.2 times as much as Sonnet 5 and matched 12 more.

    Known cost per matched vulnerability ranged from $0.47 to $21.92.

    Interpret with care: One run per configuration and application: every ratio is a single observation. Snyk Code has no per-scan model cost; its license cost is outside this study.

    Open the evidence
  4. 04

    The top two hold; the middle reorders

    Snyk Code and Opus 5 xhigh are the two highest-ranked endpoint-aware configurations under every one of the 20 alternative scoring rules; Snyk Code ranks first under 18 and Opus 5 xhigh under the other two; their order should not be read as a result.

    Line tolerance is the main driver of reordering: widening it from ±2 to ±10 lines adds 37 matches for Codex Security with GPT-5.6 Sol, 28 for Terra and 16 for Luna, at most 3 for any Claude Code configuration, and none for Snyk Code.

    Interpret with care: Rescoring shows which conclusions survive changes in the scoring rule; it does not add statistical power.

    Open the evidence

Research question

Who finds which vulnerabilities?

When security teams point AI coding agents or AI security harnesses at real applications, what do they find that deterministic SAST does not, and what do they miss that it catches?

We lead with recall, because the question practitioners ask of a scanner is whether it finds the vulnerabilities that are actually there; precision and cost are reported beside it.

Benchmark design and configurations

Nine configurations, one blind protocol

Each of the 9 configurations ran once on each of the 20 applications, for 180 successful runs. No configuration received an advisory, a class hint or the reference set.

Every result belongs to a complete harness and model configuration, never to a model alone.

Endpoint-aware configurations · Attacker-Reachable Vulnerability Recall
ConfigurationAttacker-Reachable Vulnerability Recall (macro, matched)95% intervalPrecision against the reference set (lower bound)Matched at the exact line
Snyk Code82.4% (89 of 108)74.0–90.5%20.9%85
Claude Code /security-review · Opus 5 xhigh79.3% (85 of 108)71.6–86.7%74.6%79
Claude Code /security-review · Opus 5 medium + Snyk MCP75.2% (80 of 108)68.2–82.3%82.5%74
Claude Code /security-review · Sonnet 5 xhigh67.0% (73 of 108)57.7–76.2%74.5%55
Codex Security · GPT-5.6 Luna41.2% (45 of 108)26.6–56.5%20.8%15
Codex Security · GPT-5.6 Terra30.1% (35 of 108)19.1–41.9%15.8%6
Codex Security · GPT-5.6 Sol29.2% (34 of 108)18.1–41.1%12.7%2

An empty cell means not applicable or unavailable as stated in `columns`; it never means zero.

headline.csvheadline.meta.json

DeepSec · localized recall, a different evidence requirement
ConfigurationLocalized recall (macro, matched)95% intervalPrecision against the reference set (lower bound)
DeepSec · Opus 5 xhigh37.5% (37 of 108)22.6–53.2%7.1%
DeepSec · GPT-5.6 Sol xhigh2.5% (1 of 108)0.0–7.5%10.0%

Interpretation boundary: the interval shows how much a score depends on which applications are in the corpus, not run-to-run variation.

Central result

Close at the top. Too close to separate.

A corpus of this size cannot separate the top two: a split as close as 16 to 12 would need a corpus many times larger before it could indicate a real difference, and not separating them is also not evidence that they are equal. Snyk Code ranks first in 74% of the resampled corpora.

The three Codex Security configurations matched between 34 and 45 vulnerabilities; much of that gap is about which line they report, not whether they see the vulnerability.

Horizontal bar chart of macro recall. Snyk Code 82.4% (89 of 108); Claude Code Opus 5 xhigh 79.3% (85 of 108); Opus 5 medium + Snyk MCP 75.2% (80 of 108); Sonnet 5 xhigh 67.0% (73 of 108); Codex Security GPT-5.6 Luna 41.2% (45 of 108), GPT-5.6 Terra 30.1% (35 of 108), GPT-5.6 Sol 29.2% (34 of 108). The top two cannot be separated. A separate panel shows DeepSec localized recall, a different evidence requirement: Opus 5 xhigh 37.5% (37 of 108), GPT-5.6 Sol xhigh 2.5% (1 of 108), a configuration that exported no findings in 16 of its 20 runs.Horizontal bar chart of macro recall. Snyk Code 82.4% (89 of 108); Claude Code Opus 5 xhigh 79.3% (85 of 108); Opus 5 medium + Snyk MCP 75.2% (80 of 108); Sonnet 5 xhigh 67.0% (73 of 108); Codex Security GPT-5.6 Luna 41.2% (45 of 108), GPT-5.6 Terra 30.1% (35 of 108), GPT-5.6 Sol 29.2% (34 of 108). The top two cannot be separated. A separate panel shows DeepSec localized recall, a different evidence requirement: Opus 5 xhigh 37.5% (37 of 108), GPT-5.6 Sol xhigh 2.5% (1 of 108), a configuration that exported no findings in 16 of its 20 runs.
Figure 1.Main recall results. (A) The seven endpoint-aware configurations: macro Attacker-Reachable Vulnerability Recall with the matched count; the bar is a 95% interval from resampling the 20 applications, which shows how much a score depends on which applications are in the corpus, not run-to-run variation. (B) DeepSec, scored with localized recall, on its own axis. The tick on the Claude Code rows marks the score under many-to-one credit, a sensitivity check. The top two cannot be separated: they disagree on 28 vulnerabilities, 16 vs 12 (McNemar p = 0.57, descriptive).PNG, lightPNG, darkSVG, lightSVG, dark
View exact values
Endpoint-aware configurations · Attacker-Reachable Vulnerability Recall
ConfigurationAttacker-Reachable Vulnerability Recall (macro, matched)95% intervalPrecision against the reference set (lower bound)Matched at the exact line
Snyk Code82.4% (89 of 108)74.0–90.5%20.9%85
Claude Code /security-review · Opus 5 xhigh79.3% (85 of 108)71.6–86.7%74.6%79
Claude Code /security-review · Opus 5 medium + Snyk MCP75.2% (80 of 108)68.2–82.3%82.5%74
Claude Code /security-review · Sonnet 5 xhigh67.0% (73 of 108)57.7–76.2%74.5%55
Codex Security · GPT-5.6 Luna41.2% (45 of 108)26.6–56.5%20.8%15
Codex Security · GPT-5.6 Terra30.1% (35 of 108)19.1–41.9%15.8%6
Codex Security · GPT-5.6 Sol29.2% (34 of 108)18.1–41.1%12.7%2

An empty cell means not applicable or unavailable as stated in `columns`; it never means zero.

headline.csvheadline.meta.json

View exact values
DeepSec · localized recall, a different evidence requirement
ConfigurationLocalized recall (macro, matched)95% intervalPrecision against the reference set (lower bound)
DeepSec · Opus 5 xhigh37.5% (37 of 108)22.6–53.2%7.1%
DeepSec · GPT-5.6 Sol xhigh2.5% (1 of 108)0.0–7.5%10.0%

Complementarity

Two engines reach the ceiling

Snyk Code matches 89; Opus 5 xhigh adds the 12 vulnerabilities that it matched and Snyk Code did not, reaching 101. After those two, no other configuration adds a vulnerability. Opus 5 xhigh also misses 16 vulnerabilities that Snyk Code matched, which the cumulative count does not show.

Under many-to-one credit, Snyk Code's unique coverage falls to 5 and the best pair of AI configurations reaches 93, while the combined ceiling of 101 holds.

The authorization classes illustrate the split: on the three IDOR and mass-assignment vulnerabilities, Snyk Code matched none and the Claude Code configurations matched two or three. In the other direction, Snyk Code matched all 11 open redirects, all 6 prototype-pollution vulnerabilities and all 16 path traversals.

Two panels. A staircase rises from 89 for Snyk Code to 101 after adding Claude Code Opus 5 xhigh, with no further gains from the other configurations. A diverging bar chart shows by class the 16 vulnerabilities matched only by Snyk Code and the 12 matched only by Opus 5 xhigh.Two panels. A staircase rises from 89 for Snyk Code to 101 after adding Claude Code Opus 5 xhigh, with no further gains from the other configurations. A diverging bar chart shows by class the 16 vulnerabilities matched only by Snyk Code and the 12 matched only by Opus 5 xhigh.
Figure 2.The ceiling (endpoint-aware configurations only; DeepSec excluded). (A) Greedy cover: Snyk Code matches 89; Opus 5 xhigh adds the 12 it matched and Snyk Code did not, reaching 101, the union of all seven (102 after the reference-set correction); Opus 5 xhigh also misses 16 vulnerabilities that Snyk Code matched, which the cumulative count does not show. (B) The two sets of disagreements by class; hatching marks cross-file flows.PNG, lightPNG, darkSVG, lightSVG, dark
View exact values
Greedy cover · endpoint-aware configurations only; DeepSec excluded
StepConfigurationVulnerabilities added at this stepCumulative matched
1Snyk Code8989
2Claude Code /security-review · Opus 5 xhigh12101
3Claude Code /security-review · Opus 5 medium + Snyk MCP0101
4Claude Code /security-review · Sonnet 5 xhigh0101
5Codex Security · GPT-5.6 Luna0101
6Codex Security · GPT-5.6 Terra0101
7Codex Security · GPT-5.6 Sol0101

An empty cell means not applicable or unavailable as stated in `columns`; it never means zero.

greedy_cover.csvgreedy_cover.meta.json

View exact values
Pairs of endpoint-aware configurations · DeepSec excluded
Configuration AConfiguration BMatched by eitherB adds to AA adds to BCombined findings to triage
Snyk CodeClaude Code /security-review · Opus 5 xhigh1011216539
Snyk CodeClaude Code /security-review · Opus 5 medium + Snyk MCP1001120522
Snyk CodeClaude Code /security-review · Sonnet 5 xhigh991026523
Snyk CodeCodex Security · GPT-5.6 Luna95650641
Snyk CodeCodex Security · GPT-5.6 Terra94559646
Snyk CodeCodex Security · GPT-5.6 Sol93459692
Claude Code /security-review · Opus 5 xhighCodex Security · GPT-5.6 Terra89454335
Claude Code /security-review · Opus 5 xhighCodex Security · GPT-5.6 Luna88343330
Claude Code /security-review · Opus 5 xhighCodex Security · GPT-5.6 Sol88354381
Claude Code /security-review · Opus 5 xhighClaude Code /security-review · Sonnet 5 xhigh87214212
Claude Code /security-review · Opus 5 medium + Snyk MCPClaude Code /security-review · Opus 5 xhigh8550211
Claude Code /security-review · Opus 5 medium + Snyk MCPClaude Code /security-review · Sonnet 5 xhigh85512195
Claude Code /security-review · Opus 5 medium + Snyk MCPCodex Security · GPT-5.6 Terra85550318
Claude Code /security-review · Opus 5 medium + Snyk MCPCodex Security · GPT-5.6 Luna84439313
Claude Code /security-review · Opus 5 medium + Snyk MCPCodex Security · GPT-5.6 Sol84450364
Claude Code /security-review · Sonnet 5 xhighCodex Security · GPT-5.6 Luna78533314
Claude Code /security-review · Sonnet 5 xhighCodex Security · GPT-5.6 Terra78543319
Claude Code /security-review · Sonnet 5 xhighCodex Security · GPT-5.6 Sol77443365
Codex Security · GPT-5.6 LunaCodex Security · GPT-5.6 Sol551021483
Codex Security · GPT-5.6 LunaCodex Security · GPT-5.6 Terra54919437
Codex Security · GPT-5.6 TerraCodex Security · GPT-5.6 Sol4167488

An empty cell means not applicable or unavailable as stated in `columns`; it never means zero.

pairwise_union.csvpairwise_union.meta.json

UpSet plot of the 108 reference vulnerabilities across Snyk Code, three Claude Code configurations and Codex Security pooled. One bar marks the 10 matched only by Snyk Code; another the 7 matched by none.UpSet plot of the 108 reference vulnerabilities across Snyk Code, three Claude Code configurations and Codex Security pooled. One bar marks the 10 matched only by Snyk Code; another the 7 matched by none.
Figure 3.Exact overlap of the matched sets. Each bar counts the reference vulnerabilities matched by exactly the marked combination of configurations; the three Codex Security configurations are pooled. The light bar counts the 7 matched by no endpoint-aware configuration; the dark bar the 10 only Snyk Code matched (5 under many-to-one credit). DeepSec is excluded.PNG, lightPNG, darkSVG, lightSVG, dark
View exact values
Vulnerabilities matched by exactly each combination · endpoint-aware configurations; DeepSec excluded
CombinationVulnerabilities
Snyk Code; Claude Code /security-review · Opus 5 xhigh; Claude Code /security-review · Opus 5 medium + Snyk MCP; Claude Code /security-review · Sonnet 5 xhigh; Codex Security (three configurations pooled)42
Snyk Code; Claude Code /security-review · Opus 5 xhigh; Claude Code /security-review · Opus 5 medium + Snyk MCP; Claude Code /security-review · Sonnet 5 xhigh17
Snyk Code10
No endpoint-aware configuration7
Claude Code /security-review · Opus 5 xhigh; Claude Code /security-review · Opus 5 medium + Snyk MCP; Claude Code /security-review · Sonnet 5 xhigh; Codex Security (three configurations pooled)7
Snyk Code; Claude Code /security-review · Opus 5 xhigh; Claude Code /security-review · Opus 5 medium + Snyk MCP7
Snyk Code; Codex Security (three configurations pooled)4
Snyk Code; Claude Code /security-review · Opus 5 xhigh; Claude Code /security-review · Opus 5 medium + Snyk MCP; Codex Security (three configurations pooled)3
Claude Code /security-review · Opus 5 xhigh; Claude Code /security-review · Opus 5 medium + Snyk MCP; Claude Code /security-review · Sonnet 5 xhigh2
Snyk Code; Claude Code /security-review · Opus 5 xhigh2
Claude Code /security-review · Opus 5 xhigh; Claude Code /security-review · Sonnet 5 xhigh1
Claude Code /security-review · Opus 5 xhigh; Claude Code /security-review · Opus 5 medium + Snyk MCP1
Claude Code /security-review · Opus 5 xhigh; Claude Code /security-review · Opus 5 medium + Snyk MCP; Codex Security (three configurations pooled)1
Snyk Code; Claude Code /security-review · Sonnet 5 xhigh1
Snyk Code; Claude Code /security-review · Sonnet 5 xhigh; Codex Security (three configurations pooled)1
Snyk Code; Claude Code /security-review · Opus 5 xhigh; Claude Code /security-review · Sonnet 5 xhigh1
Snyk Code; Claude Code /security-review · Opus 5 xhigh; Claude Code /security-review · Sonnet 5 xhigh; Codex Security (three configurations pooled)1

An empty cell means not applicable or unavailable as stated in `columns`; it never means zero.

overlap_intersections.csvoverlap_intersections.meta.json

Where each engine wins

Within one file and across files

Within a single file Snyk Code has the highest recall: 40 of the 46 same-file vulnerabilities, against 32 for Opus 5 xhigh. Across files the two are close: 49 of 62 and 53.

For the Claude Code configurations, the blind spots are about which classes they report, not about navigating a codebase.

One cell is hard for every engine: cross-file cross-site scripting, where no endpoint-aware configuration matched more than 4 of the 7. In 5 of those 7 the data flows from server-side code to a browser page; Snyk Code matched none of the 5 and Opus 5 xhigh 2.

Snyk Code's lower cross-file recall comes largely from this cell: with cross-site scripting removed, its cross-file and same-file recall are essentially equal (47 of 55 and 29 of 34).

Three panels with one row per endpoint-aware configuration. Same-file and cross-file recall: Snyk Code 40/46 and 49/62; Opus 5 xhigh 32/46 and 53/62; Opus 5 med + MCP 30/46 and 50/62; Sonnet 5 xhigh 27/46 and 46/62; Codex Security GPT-5.6 Luna 14/46 and 31/62, Terra 14/46 and 21/62, Sol 12/46 and 22/62.Three panels with one row per endpoint-aware configuration. Same-file and cross-file recall: Snyk Code 40/46 and 49/62; Opus 5 xhigh 32/46 and 53/62; Opus 5 med + MCP 30/46 and 50/62; Sonnet 5 xhigh 27/46 and 46/62; Codex Security GPT-5.6 Luna 14/46 and 31/62, Terra 14/46 and 21/62, Sol 12/46 and 22/62.
Figure 4.Where each engine wins, by flow shape. (A) Recall on the 46 same-file and 62 cross-file vulnerabilities. (B) What each configuration's same-file misses are. (C) Cross-file cross-site scripting, the one cell where every configuration matched at most 4 of 7. Rows are the same configuration in every panel.PNG, lightPNG, darkSVG, lightSVG, dark
View exact values
Same-file and cross-file vulnerabilities · endpoint-aware configurations
ConfigurationSame-file matchedCross-file matchedSame-file missesMisses in classes the built-in review policy excludes or restrictsMisses in cross-site scripting
Snyk Code40 of 4649 of 62611
Claude Code /security-review · Opus 5 xhigh32 of 4653 of 621483
Claude Code /security-review · Opus 5 medium + Snyk MCP30 of 4650 of 621693
Claude Code /security-review · Sonnet 5 xhigh27 of 4646 of 6219114
Codex Security · GPT-5.6 Luna14 of 4631 of 6232119
Codex Security · GPT-5.6 Terra14 of 4621 of 6232118
Codex Security · GPT-5.6 Sol12 of 4622 of 6234127

An empty cell means not applicable or unavailable as stated in `columns`; it never means zero.

flow_shape.csvflow_shape.meta.json

The cost of combining

A larger queue to triage

Snyk Code and Opus 5 xhigh together report 539 findings for their 101 matches. 336 of Snyk Code's 425 findings matched no reference vulnerability; 87% of those are in classes outside the 14 scored classes, led by hardcoded credentials (195).

Counting only reports in the scored classes, precision is 69% (89 of 129) for Snyk Code against 81% (85 of 105) for Opus 5 xhigh.

Stacked bars of findings outside the reference set per endpoint-aware configuration, led by Snyk Code's 336, most in classes outside the 14 scored classes; a second panel compares precision against the reference set with precision over in-taxonomy reports.Stacked bars of findings outside the reference set per endpoint-aware configuration, led by Snyk Code's 336, most in classes outside the 14 scored classes; a second panel compares precision against the reference set with precision over in-taxonomy reports.
Figure 5.The triage cost. (A) Findings that matched no reference vulnerability, by category; they are outside the reference set and not proven false. (B) Precision against the reference set and over reports in the 14 scored classes only; both are lower bounds. DeepSec is omitted.PNG, lightPNG, darkSVG, lightSVG, dark
View exact values
Findings and precision against the reference set · endpoint-aware configurations; DeepSec omitted
ConfigurationFindingsMatchedPrecision against the reference set (lower bound)
Snyk Code4258920.9%
Claude Code /security-review · Opus 5 xhigh1148574.6%
Claude Code /security-review · Opus 5 medium + Snyk MCP978082.5%
Claude Code /security-review · Sonnet 5 xhigh987374.5%
Codex Security · GPT-5.6 Luna2164520.8%
Codex Security · GPT-5.6 Terra2213515.8%
Codex Security · GPT-5.6 Sol2673412.7%

An empty cell means not applicable or unavailable as stated in `columns`; it never means zero.

headline.csvheadline.meta.json

Findings outside the reference set are not proven false; both precision measures are lower bounds.

Findings outside the reference set

Real vulnerabilities the answer key missed

23 clusters of findings outside the reference set were reported by at least three configurations from at least two tool families. We reviewed the 15 of them that Snyk Code did not report and that fall in a scored class or are CSRF: thirteen were real, one was borderline and one was judged to be by design; none was noise.

In this run, AI configurations found at least seven real vulnerabilities in scored classes that the reference set did not contain and Snyk Code did not report. All seven were reproduced live.

Codex Security configurations reported all seven, the Claude Code configurations six and DeepSec three; the consensus filter gives more weight to the family with three configurations, so these counts do not measure how broadly each family discovers issues.

They follow data where a first-party taint model typically ends: through library semantics and across the server-to-browser hop.

Each candidate had a code-level first pass and an independent skeptical second pass, both by AI agents, and a final call by an author; the consensus filter favors issues several tools can see. These findings are never added to the reference set or the recall results, and they are a lower-bound count, not a miss rate for any engine.

Harness policy

A built-in policy that suppresses real classes

The built-in /security-review prompt in Claude Code 2.1.251, the version that ran, excludes ReDoS and asks that prototype pollution and open redirect be reported only at extremely high confidence. These classes were chosen from the prompt text, before looking at results.

No Claude Code configuration reported a single ReDoS finding anywhere in the three applications that contain one. For the 11 open redirects, counting a finding of the class anywhere in the application, the Claude Code configurations reported 2 to 6 of the 11 and each Codex Security configuration reported 10 of the 11, though Codex Security was credited for only 4 or 5 of the 11 open redirects; Snyk Code reported and was credited for all 11.

Prototype pollution is weaker evidence: Opus 5 at both effort settings reported all 6 and was credited for 4.

Prompt-swap control (supplementary, 19 of the 20 applications, single run): matched 16 of the 20 vulnerabilities in the classes the policy excludes or restricts, against 9 with /security-review, with little change elsewhere.

Grouped bars of credited recall for the seven endpoint-aware configurations. In the 20 policy-class vulnerabilities Snyk Code matched 18 and Opus 5 xhigh 9; the supplementary prompt-swap control (single run, 19 of the 20 applications), outlined, reached 16 of the 20. A second panel shows, per class, credited and reported counts.Grouped bars of credited recall for the seven endpoint-aware configurations. In the 20 policy-class vulnerabilities Snyk Code matched 18 and Opus 5 xhigh 9; the supplementary prompt-swap control (single run, 19 of the 20 applications), outlined, reached 16 of the 20. A second panel shows, per class, credited and reported counts.
Figure 6.Classes the built-in review policy excludes or restricts. The Claude Code /security-review prompt (version 2.1.251) excludes ReDoS and asks that prototype pollution and open redirect be reported only at extremely high confidence; these classes were chosen from the prompt text, not from the data. (A) Credited recall on the 20 vulnerabilities in those classes, the 7 cross-site scripting vulnerabilities in React and Angular applications, and the 81 others. (B) Per class, credited against reported anywhere in the application. The dashed outline is the supplementary prompt-swap control (19 applications, single run), shown beside the main results and not merged into them.PNG, lightPNG, darkSVG, lightSVG, dark
View exact values
Credited recall by subset · endpoint-aware configurations, main runs
ConfigurationClasses the built-in review policy excludes or restrictsXSS in React/Angular applicationsAll other vulns
Snyk Code18 of 204 of 767 of 81
Claude Code /security-review · Opus 5 xhigh9 of 205 of 771 of 81
Claude Code /security-review · Opus 5 medium + Snyk MCP5 of 205 of 770 of 81
Claude Code /security-review · Sonnet 5 xhigh6 of 204 of 763 of 81
Codex Security · GPT-5.6 Luna6 of 203 of 736 of 81
Codex Security · GPT-5.6 Terra5 of 201 of 729 of 81
Codex Security · GPT-5.6 Sol4 of 201 of 729 of 81

An empty cell means not applicable or unavailable as stated in `columns`; it never means zero.

policy_class_recall.csvpolicy_class_recall.meta.json

View exact values
Per class, credited against reported anywhere in the application · main runs
ConfigurationClassCreditedReported anywhere in the application
Snyk Codeopen redirect11 of 1111 of 11
Snyk Codeprototype pollution6 of 66 of 6
Snyk CodeReDoS1 of 32 of 3
Claude Code /security-review · Opus 5 xhighopen redirect5 of 116 of 11
Claude Code /security-review · Opus 5 xhighprototype pollution4 of 66 of 6
Claude Code /security-review · Opus 5 xhighReDoS0 of 30 of 3
Claude Code /security-review · Opus 5 medium + Snyk MCPopen redirect1 of 112 of 11
Claude Code /security-review · Opus 5 medium + Snyk MCPprototype pollution4 of 66 of 6
Claude Code /security-review · Opus 5 medium + Snyk MCPReDoS0 of 30 of 3
Claude Code /security-review · Sonnet 5 xhighopen redirect4 of 115 of 11
Claude Code /security-review · Sonnet 5 xhighprototype pollution2 of 63 of 6
Claude Code /security-review · Sonnet 5 xhighReDoS0 of 30 of 3
Codex Security · GPT-5.6 Lunaopen redirect5 of 1110 of 11
Codex Security · GPT-5.6 Lunaprototype pollution1 of 63 of 6
Codex Security · GPT-5.6 LunaReDoS0 of 30 of 3
Codex Security · GPT-5.6 Terraopen redirect5 of 1110 of 11
Codex Security · GPT-5.6 Terraprototype pollution0 of 61 of 6
Codex Security · GPT-5.6 TerraReDoS0 of 33 of 3
Codex Security · GPT-5.6 Solopen redirect4 of 1110 of 11
Codex Security · GPT-5.6 Solprototype pollution0 of 66 of 6
Codex Security · GPT-5.6 SolReDoS0 of 31 of 3

An empty cell means not applicable or unavailable as stated in `columns`; it never means zero.

policy_class_detail.csvpolicy_class_detail.meta.json

View exact values
Supplementary prompt-swap control · single run · never merged into the main results
SubsetSupplementary prompt-swap control (Opus 5 xhigh, default audit prompt)Main run: Claude Code /security-review · Opus 5 xhigh, same applications
Classes the built-in review policy excludes or restricts16 of 209 of 20
All other non-React/Angular71 of 7869 of 78
Open redirect9 of 115 of 11
Prototype pollution5 of 64 of 6
ReDoS2 of 30 of 3

An empty cell means not applicable or unavailable as stated in `columns`; it never means zero.

policy_control_supplementary.csvpolicy_control_supplementary.meta.json

The control is shown beside the main results and never merged into them; changes on the model provider's side cannot be excluded.

Localization

The right function, a line above the dangerous call

Codex Security with GPT-5.6 Sol located 80 of the 108 vulnerabilities at file level, but 34 under the two-line rule. When it missed a curated sink, its nearest reported location in the same file sat above the curated line in 39 of the 41 cases where that location was not itself a source, a median of six lines above.

A harness can detect a vulnerability and still hand the developer the wrong line. Line-level evidence is what makes a finding actionable.

Histogram of line offsets for 57 missed sinks; most bars sit left of zero, meaning the reported line is above the dangerous call; 39 of 41 non-source locations are above it.Histogram of line offsets for 57 missed sinks; most bars sit left of zero, meaning the reported line is above the dangerous call; 39 of 41 non-source locations are above it.
Figure 7.Codex Security with GPT-5.6 Sol: for each of the 57 missed curated sinks with a reported location in the same file, the offset of the nearest reported location from the sink line (negative means above). Among the 41 locations that are not themselves a curated source, 39 lie above the sink, median six lines. The shaded band is the two-line tolerance.PNG, lightPNG, darkSVG, lightSVG, dark
View exact values
Codex Security · GPT-5.6 Sol · missed curated sinks with a reported location in the same file
OffsetMissed curated sinks
Offset in linesReported location is a curated sourceReported location is not a curated source
≤−2101
−20…−1118
−10…−6915
−5…−3615
≥+302

Offset: Reported line minus curated sink line; negative = above the sink.

An empty cell means not applicable or unavailable as stated in `columns`; it never means zero.

sol_sink_offsets.csvsol_sink_offsets.meta.json

Line chart of vulnerabilities matched against line tolerance. Snyk Code and the Claude Code configurations stay nearly flat; the Codex Security configurations rise steeply, GPT-5.6 Sol from 34 at two lines to 80 at file level. DeepSec is on a separate axis.Line chart of vulnerabilities matched against line tolerance. Snyk Code and the Claude Code configurations stay nearly flat; the Codex Security configurations rise steeply, GPT-5.6 Sol from 34 at two lines to 80 at file level. DeepSec is on a separate axis.
Figure 8.Matches against line tolerance. (A) The seven endpoint-aware configurations under the endpoint rule, from exact line to file level; only the two-line setting is used for results, the others are counterfactual rescorings of the same findings. (B) DeepSec under its own localized rule, on a separate axis.PNG, lightPNG, darkSVG, lightSVG, dark
View exact values
Vulnerabilities matched against line tolerance · endpoint rule; DeepSec excluded
ConfigurationLine tolerance
ConfigurationExact line±1 lines±2 lines (used for results)±3 lines±5 lines±10 lines±20 linesAny line in the right file
Snyk Code8588898989898989
Claude Code /security-review · Opus 5 xhigh7984858686868689
Claude Code /security-review · Opus 5 medium + Snyk MCP7479808181818184
Claude Code /security-review · Sonnet 5 xhigh5571737575767776
Codex Security · GPT-5.6 Luna1532455055616466
Codex Security · GPT-5.6 Terra620353947636669
Codex Security · GPT-5.6 Sol218344354717780

An empty cell means not applicable or unavailable as stated in `columns`; it never means zero.

tolerance_sweep.csvtolerance_sweep.meta.json

A pre-model ceiling

DeepSec's file selection fixes a recall ceiling before the model runs

DeepSec's model-free scan stage queued at least one file from the curated flow for only 56 of the 108 vulnerabilities: all 12 Java vulnerabilities and 23 of 30 in JavaScript and TypeScript, but 11 of 32 in Go and 10 of 34 in Python.

Conditional on selection, the Opus 5 backend's localized recall was 37 of 56 and the GPT-5.6 Sol backend's 1 of 56. The GPT-5.6 Sol backend exported no findings in 16 of its 20 runs; the records cannot tell the backend's behavior inside DeepSec from an integration problem, so this row describes the configuration as executed and says nothing about the model.

Funnel from 108 reference vulnerabilities to 56 with a flow file selected, then to 37 localized matches for the Opus 5 backend and 1 for the GPT-5.6 Sol backend, which exported no findings in 16 of its 20 runs; a second panel shows candidate coverage by language.Funnel from 108 reference vulnerabilities to 56 with a flow file selected, then to 37 localized matches for the Opus 5 backend and 1 for the GPT-5.6 Sol backend, which exported no findings in 16 of its 20 runs; a second panel shows candidate coverage by language.
Figure 9.Where DeepSec 2.3.7 loses the reference set. (A) Of 108 reference vulnerabilities, 56 had a flow file among the scan stage's candidates (model-free, identical for both backends) and 44 had their sink file there; localized matches follow for each backend. (B) Candidate coverage by language family. Localized recall is not comparable with Attacker-Reachable Vulnerability Recall.PNG, lightPNG, darkSVG, lightSVG, dark
View exact values
DeepSec · from the reference set through the scan stage to localized matches
StageVulnerabilities
Reference vulnerabilities108
A flow file among the scan stage's candidates56 of 108
The sink file among the candidates (vulnerabilities with a labeled sink)44 of 107
Localized match, Opus 5 xhigh backend37 of 56
Localized match, GPT-5.6 Sol backend1 of 56

Localized recall is DeepSec's primary metric and is not comparable with Attacker-Reachable Vulnerability Recall.

An empty cell means not applicable or unavailable as stated in `columns`; it never means zero.

deepsec_funnel.csvdeepsec_funnel.meta.json

View exact values
DeepSec · a flow file among the scan stage's candidates, by language family
Language familyVulnerabilities
Go11 of 32
JavaScript / TypeScript23 of 30
Python10 of 34
Java12 of 12

Localized recall is DeepSec's primary metric and is not comparable with Attacker-Reachable Vulnerability Recall.

Efficiency

What coverage costs

Per matched vulnerability, wall time was 5 seconds for Snyk Code and between 107 and 942 seconds for the endpoint-aware AI configurations.

Known cost per matched vulnerability ranged from $0.47 (Codex Security with GPT-5.6 Luna) to $21.92 (Codex Security with GPT-5.6 Sol). The recall-versus-cost frontier consists of Codex Security with GPT-5.6 Luna and the three Claude Code configurations.

Per matched vulnerability, the Claude Code configurations reported 1.21 to 1.34 findings, Snyk Code 4.78 and the Codex Security configurations 4.80 to 7.85: a measure of how much a team would review, not how much of it is noise.

The Snyk MCP configuration matched 80 vulnerabilities, every one of which Opus 5 xhigh also matched, at 0.35 times the cost. Effort, MCP access and the prompt all differ, so this is a cheaper configuration that kept 80 of the 85, not an effect of grounding.

Two scatter plots of macro recall. Against cost, Snyk Code sits in a separate no-per-scan-model-cost column; arrows mark two same-harness contrasts: Codex Security GPT-5.6 Sol cost 35.0 times GPT-5.6 Luna for 11 fewer matches, and Claude Code Opus 5 cost 4.2 times Sonnet 5 for 12 more. Against wall time, Snyk Code took 8 minutes in total against at least 2.4 hours for the AI configurations.Two scatter plots of macro recall. Against cost, Snyk Code sits in a separate no-per-scan-model-cost column; arrows mark two same-harness contrasts: Codex Security GPT-5.6 Sol cost 35.0 times GPT-5.6 Luna for 11 fewer matches, and Claude Code Opus 5 cost 4.2 times Sonnet 5 for 12 more. Against wall time, Snyk Code took 8 minutes in total against at least 2.4 hours for the AI configurations.
Figure 10.Coverage against cost and time (endpoint-aware configurations; DeepSec excluded). (A) Macro recall against mean known cost per application, log scale; Snyk Code has no per-scan model cost and is shown in a separate column, not on the dollar axis. Ringed points form the recall-versus-cost frontier. (B) Macro recall against mean wall time per application, log scale. One run per configuration and application; every ratio is a single observation.PNG, lightPNG, darkSVG, lightSVG, dark
View exact values
Resources per matched vulnerability · endpoint-aware configurations
ConfigurationWall time, totalWall time per matched vulnerability (s)Cost statusKnown cost, total (USD)Known cost per matched vulnerability (USD)Tokens per matched vulnerability (millions)Findings per matched vulnerability
Snyk Code8 min5no per-scan model cost———4.78
Claude Code /security-review · Opus 5 xhigh5.6 h236known164.531.941.031.34
Claude Code /security-review · Opus 5 medium + Snyk MCP2.4 h107known57.440.720.351.21
Claude Code /security-review · Sonnet 5 xhigh3.1 h153known39.560.540.721.34
Codex Security · GPT-5.6 Luna5.4 h433known21.320.4724.084.8
Codex Security · GPT-5.6 Terra4.9 h508known207.995.9424.426.31
Codex Security · GPT-5.6 Sol8.9 h942known745.1721.9242.117.85

One run per configuration and application: every ratio is a single observation. Snyk Code has no per-scan model cost; its license cost is outside this study.

An empty cell means not applicable or unavailable as stated in `columns`; it never means zero.

efficiency.csvefficiency.meta.json

View exact values
DeepSec · per localized-recall match; never compared with the endpoint-aware configurations
ConfigurationWall time, totalWall time per matched vulnerability (s)Cost statusKnown cost, total (USD)Known cost per matched vulnerability (USD)Tokens per matched vulnerability (millions)
DeepSec · Opus 5 xhigh17.8 h1733known355.179.603.69
DeepSec · GPT-5.6 Sol xhigh3.3 h11752unavailable——12.99
Four bar panels per endpoint-aware configuration. Wall time per match: Snyk Code 5 s, AI configurations 107 to 942 s. Known cost per match: $0.47 to $21.92; Snyk Code has no per-scan model cost. Tokens per match: Claude Code 0.35 to 1.03 million, Codex Security 24 to 42 million. Findings per match: Claude Code 1.21 to 1.34, Snyk Code 4.78, Codex Security 4.80 to 7.85.Four bar panels per endpoint-aware configuration. Wall time per match: Snyk Code 5 s, AI configurations 107 to 942 s. Known cost per match: $0.47 to $21.92; Snyk Code has no per-scan model cost. Tokens per match: Claude Code 0.35 to 1.03 million, Codex Security 24 to 42 million. Findings per match: Claude Code 1.21 to 1.34, Snyk Code 4.78, Codex Security 4.80 to 7.85.
Figure 11.Resources per matched reference vulnerability: totals over the 20 applications divided by the number matched. (A) Wall time, log scale. (B) Known dollar cost, log scale; Snyk Code has no per-scan model cost. (C) Tokens, log scale; provider accounting differs, so compare within a family. (D) Reported findings per matched vulnerability, a measure of review volume.PNG, lightPNG, darkSVG, lightSVG, dark

One run per configuration and application: every ratio is a single observation. Cost is known cost only ($1,594.08 across the main matrix); an unavailable cost is never shown as zero. Snyk Code has no per-scan model cost; its license cost is outside this study. Tokens compare only within a provider family.

Robustness

Which conclusions depend on the scoring rule

Rank agreement with the rule used for all results ranges from a Kendall τ of 1.00 down to 0.33; the top four change membership under 7 of the 20 alternatives.

Leaving out one application at a time changes some rank in 4 of the 20 cases and the top rank in none.

In two supplementary repetitions, Snyk Code returned byte-identical findings on all 20 applications, and the same 89 matches each time. The AI configurations ran once per application.

Grid of ranks, one row per scoring rule. Snyk Code and Opus 5 xhigh hold the top two ranks in every row, with Snyk Code first under 18 of the 20 alternatives and Opus 5 xhigh first under the other two; the middle ranks reorder under lenient location rules; Kendall's τ ranges from 1.00 down to 0.33.Grid of ranks, one row per scoring rule. Snyk Code and Opus 5 xhigh hold the top two ranks in every row, with Snyk Code first under 18 of the 20 alternatives and Opus 5 xhigh first under the other two; the middle ranks reorder under lenient location rules; Kendall's τ ranges from 1.00 down to 0.33.
Figure 12.Rank of each endpoint-aware configuration by macro recall under the scoring rule used for all results (outlined row) and 20 alternative rules, grouped as line tolerance, endpoint roles, matching and credit, and location-only or type-only rules. The right column gives Kendall's τ against the outlined row. DeepSec is excluded because it reports no endpoint roles.PNG, lightPNG, darkSVG, lightSVG, dark
View exact values
Rank of each endpoint-aware configuration by macro recall under each scoring rule · DeepSec excluded
Scoring ruleRank by macro recallAgreement
Scoring ruleSnyk CodeClaude Code /security-review · Opus 5 xhighClaude Code /security-review · Opus 5 medium + Snyk MCPClaude Code /security-review · Sonnet 5 xhighCodex Security · GPT-5.6 LunaCodex Security · GPT-5.6 TerraCodex Security · GPT-5.6 SolKendall's τ
Endpoint rule, ±2 lines (used for all results)12345671.00
Endpoint rule, exact line12345671.00
Endpoint rule, ±112345671.00
Endpoint rule, ±312345760.90
Endpoint rule, ±512346750.81
Endpoint rule, ±1012347650.71
Endpoint rule, ±2012357640.62
Endpoint rule, right file, any line21357640.52
Source and sink both required, exact line12345671.00
Source and sink both required, ±212345671.00
Sink required, ±212345671.00
Source required, ±212346750.81
Maximum matching instead of greedy12345671.00
No type aliases12345671.00
Many-to-one credit12345671.00
Any location, ±212356740.71
Any location, exact line12345760.90
Any location, ±512357640.62
Any location, ±1012357640.62
Any location, right file, any line21376540.33
Vulnerability type only12476530.33

Rank: Rank by macro recall under that rule (1 = highest). Kendall's τ: Rank agreement with the rule used for all results (first row).

An empty cell means not applicable or unavailable as stated in `columns`; it never means zero.

rank_stability.csvrank_stability.meta.json

Operational meaning

Practical implications for security teams

  1. 01

    Run deterministic SAST and AI review together

    In this study neither was sufficient alone, and together they reached the ceiling of what all comparable configurations found, at the price of a larger triage queue. Deterministic SAST was measured on Snyk Code, the only SAST engine evaluated.

  2. 02

    Read the harness as part of what you buy

    A built-in policy, a localization habit or a pre-model file filter shapes what reaches the report.

  3. 03

    Judge spend by coverage, not by price

    A more expensive model did not reliably buy more coverage.

How 2.0 differs from JS 1.0

Two releases, two questions

VulnBench 2.0 takes the next steps JS 1.0 named: application-scale fixtures, an independent ground truth and stricter scoring. Its numbers are not comparable with JS 1.0's.

Snyk VulnBench JS 1.0 and Snyk VulnBench 2.0, aspect by aspect
AspectSnyk VulnBench JS 1.0Snyk VulnBench 2.0
QuestionCan LLMs find the same bugs twice? Repeatability and agreement.What do AI security agents find that SAST does not, and what do they miss?
FixturesTen small JavaScript and Express projects20 open-source web applications in Go, JavaScript/TypeScript, Python and Java
Reference setSnyk Code's findings define the reference setCurated, attacker-reachable vulnerabilities with annotated source-to-sink flows and author-verified exploits; not derived from any tool's findings
MatchingSame vulnerability type; file and line not requiredClass plus source and sink, each within two lines; one finding credited to at most one vulnerability
SystemsClaude configurations through one agentic harness, plus Snyk Code reproducing its own referenceProductized harnesses: Snyk Code, Claude Code /security-review, Codex Security, DeepSec
RepetitionsFive identical runs per configuration and projectOne run per configuration and application; Snyk Code repeated twice more in supplementary runs
Headline metricSnyk-reference F1 and recurrenceAttacker-Reachable Vulnerability Recall (macro, with the matched count); DeepSec scored with localized recall

Snyk Code's 100% in JS 1.0 is reproduction of the reference set it defined; in 2.0 Snyk Code is scored against an independent reference set like every other configuration. Never compare a 2.0 number with a JS 1.0 number.

Limitations

What the protocol cannot establish

  • One run per configuration and application: run-to-run variation of the AI configurations is unmeasured, and every difference between them is a single observation.
  • All authors are employees of Snyk, which develops Snyk Code. The Snyk Code product owner is a co-author, and his team shipped a rule update for one of the findings outside the reference set; all main results are reported as run, before that update.
  • Reference vulnerabilities were partly seeded with OpenAI and Anthropic models, which could favor the AI configurations; some seeding prompts included vulnerable-pattern examples from a Snyk database, and five applications had been used in Snyk demonstrations, which could favor Snyk Code. The net effect is unknown.
  • Without the five demonstration applications, Snyk Code matches 74 and Opus 5 xhigh 72 of the remaining 87 vulnerabilities, and they remain the top two.
  • The reference set can omit real vulnerabilities; measured precision is a lower bound. One known defect makes one vulnerability unmatchable; correcting it raises Claude Code /security-review with Opus 5 medium + Snyk MCP from 80 to 81 and Opus 5 xhigh from 85 to 86, and the union of all seven to 102.
  • The two-line tolerance, the endpoint rule and one-to-one credit are choices, not facts about the tools; the robustness section shows which conclusions depend on them.
  • Snyk Code is the only SAST engine evaluated; statements about deterministic SAST are measured on Snyk Code and may not hold for other engines.
  • Results belong to the configurations as run, including versions and modes; the design cannot separate harness from model.
  • Cost, token and time figures come from single runs; unavailable costs are excluded, never treated as zero.
  • Class names follow Snyk Code's vocabulary; findings labeled outside it could go uncredited. The measured effect is small: removing the reference set's alternative class names changes one match.
  • The origin of each vulnerability, and which model seeded which application, were not recorded.
  • The archive does not record the Snyk CLI version (the operator reported it after the run), and the Snyk MCP server ran unpinned, so its version is inferred; exact re-execution of those rows is bounded, re-scoring is not.
  • Codex Security ran in its standard full-repository mode, not its deep mode, and its scans reported partial coverage on 7, 6 and 3 of the 20 applications for GPT-5.6 Luna, Terra and Sol; partial scans are scored as returned.
  • Five failed attempts preceded the scored success in 4 of the 180 runs; only the final attempt is scored.
  • Several applications are intentionally vulnerable demonstrations, many vulnerabilities were seeded, and Java is thin; how the results carry over to production code bases is untested.

Continue the research trail

Inspect the evidence behind the narrative