An attacker-reachable recall study under one blind protocol

01 · Benchmark design

108 vulnerabilities in 20 applications

The corpus contains 108 reference vulnerabilities, between 2 and 10 per application (median 5). For 62 of them the curated flow from source to sink spans more than one file; the other 46 stay within a single file.

The vulnerabilities fall into 14 classes, led by cross-site scripting (19), path traversal (16), command injection (13) and SQL injection (13). Across the corpus there are 447 authored flow locations: 131 sources, 118 sinks and 198 unlabeled intermediate steps.

Sixteen applications are third-party projects; four were written by Snyk employees for earlier work and reused here.

02 · Reference set

An independent, attacker-reachable reference set

Reference vulnerabilities either already existed in intentionally vulnerable demonstration applications or were seeded by the authors with an AI coding assistant, using OpenAI and Anthropic models. Every one was then curated with an annotated source-to-sink flow and an exploit proof of concept verified by the authors.

Attacker-reachable means that input an outside attacker controls reaches a dangerous operation without being neutralized.

The reference set is not derived from any tool's findings. Snyk Code was used during curation only as a format and class-naming reference.

03 · Configurations

Nine configurations

Snyk Code runs through its command-line interface with no prompt. Claude Code runs its built-in /security-review command with three model and effort settings, one with the Snyk MCP server. Codex Security runs in its standard full-repository mode with GPT-5.6 Luna, GPT-5.6 Terra or GPT-5.6 Sol. DeepSec runs with an Opus 5 or a GPT-5.6 Sol backend.

In the MCP configuration the agent must call snyk_code_scan exactly once and, by construction, receives Snyk Code's findings for the application (the MCP responses were not recorded); it does not receive coverage information about what Snyk Code analyzed.

Versions and how each is known are recorded per row; vendors' built-in prompts are paraphrased, not reproduced.

04 · Protocol

Blind, independent runs

No configuration receives an advisory, a vulnerability description, a class hint or the reference set. Every run starts in a fresh, isolated workspace with only the application source.

Each of the 9 configurations ran once on each of the 20 applications, for 180 successful runs.

05 · Scoring

The endpoint rule

A finding matches when its class equals the reference class or a listed alternative and its reported locations meet the endpoint rule, with a tolerance of two lines: for a flow with one curated location, a match to that location is enough; for two, a match to either labeled endpoint; for three or more, distinct reported locations must match both a labeled source and a labeled sink.

Of the 62 cross-file flows, 61 fall under the strict three-or-more rule, against 29 of the 46 same-file flows.

Matching is one-to-one: a finding can be credited to at most one vulnerability.

Why two lines: the tolerance absorbs a tool and an annotator anchoring different lines of the same statement. Under exact-line matching Snyk Code matches 85 instead of 89, and the ranking of the seven comparable configurations is unchanged.

DeepSec exports files and lines but no source or sink roles, so it is scored with localized recall: a class match plus any reported location within two lines of any curated flow location.

06 · Metric definitions

Each value answers a narrow question

Attacker-Reachable Vulnerability Recall

The share of reference vulnerabilities a configuration matched under the endpoint rule.

Unit: percent · Aggregation: Macro average over the 20 applications, with the pooled matched count beside itPooled (micro) and macro recall rank the configurations identically.

Localized recall (DeepSec)

A class match plus any reported location within two lines of any curated flow location.

Unit: percent · Aggregation: Macro average over the 20 applicationsA different evidence requirement; never ranked or combined with Attacker-Reachable Vulnerability Recall.

Precision against the reference set

The share of a configuration's findings that matched a reference vulnerability.

Unit: percent · Aggregation: Pooled over all findingsA lower bound, because the reference set may omit real vulnerabilities.

95% interval

The middle 95% of macro recall over 4,000 resamples of the 20 applications (seed 20260929).

Unit: percent · Aggregation: Resampling applications with replacementShows sensitivity to which applications are in the corpus, not run-to-run variation.

Paired comparison

McNemar's test looks only at the vulnerabilities one configuration found and the other missed.

Unit: p-value · Aggregation: Per pair of configurationsDescriptive and unadjusted; no difference is described as significant.

Wall time

Elapsed time per run, including each harness's pipeline and the shared gateway.

Unit: seconds or hours · Aggregation: Total over 20 applications, or per matched vulnerabilityMeasures each configuration as deployed, not model speed alone.

Known cost

The cost recorded by the harness for each run.

Unit: USD · Aggregation: Total over 20 applications divided by matched vulnerabilitiesSnyk Code has no per-scan model cost; an unavailable cost is never shown as zero.

Tokens

Tokens recorded by each harness for its runs.

Unit: tokens · Aggregation: Per matched vulnerabilityProviders account for cached input differently; compare within a family.

Findings per matched vulnerability

Reported findings divided by matched vulnerabilities.

Unit: ratio · Aggregation: Totals over 20 applicationsA measure of review volume, not of noise.

07 · Supplementary runs

Supplementary runs

Three supplementary arms ran about a month after the main runs with the same tasks, reference set and prompts: an equal-effort comparison for the MCP configuration, a prompt-swap control for the review policy, and two repeats of Snyk Code.

They are reported beside the main results and are never merged into the main tables, unions or rankings.

08 · Limitations

What the protocol cannot establish

• One run per configuration and application: run-to-run variation of the AI configurations is unmeasured, and every difference between them is a single observation.
• All authors are employees of Snyk, which develops Snyk Code. The Snyk Code product owner is a co-author, and his team shipped a rule update for one of the findings outside the reference set; all main results are reported as run, before that update.
• Reference vulnerabilities were partly seeded with OpenAI and Anthropic models, which could favor the AI configurations; some seeding prompts included vulnerable-pattern examples from a Snyk database, and five applications had been used in Snyk demonstrations, which could favor Snyk Code. The net effect is unknown.
• Without the five demonstration applications, Snyk Code matches 74 and Opus 5 xhigh 72 of the remaining 87 vulnerabilities, and they remain the top two.
• The reference set can omit real vulnerabilities; measured precision is a lower bound. One known defect makes one vulnerability unmatchable; correcting it raises Claude Code /security-review with Opus 5 medium + Snyk MCP from 80 to 81 and Opus 5 xhigh from 85 to 86, and the union of all seven to 102.
• The two-line tolerance, the endpoint rule and one-to-one credit are choices, not facts about the tools; the robustness section shows which conclusions depend on them.
• Snyk Code is the only SAST engine evaluated; statements about deterministic SAST are measured on Snyk Code and may not hold for other engines.
• Results belong to the configurations as run, including versions and modes; the design cannot separate harness from model.
• Cost, token and time figures come from single runs; unavailable costs are excluded, never treated as zero.
• Class names follow Snyk Code's vocabulary; findings labeled outside it could go uncredited. The measured effect is small: removing the reference set's alternative class names changes one match.
• The origin of each vulnerability, and which model seeded which application, were not recorded.
• The archive does not record the Snyk CLI version (the operator reported it after the run), and the Snyk MCP server ran unpinned, so its version is inferred; exact re-execution of those rows is bounded, re-scoring is not.
• Codex Security ran in its standard full-repository mode, not its deep mode, and its scans reported partial coverage on 7, 6 and 3 of the 20 applications for GPT-5.6 Luna, Terra and Sol; partial scans are scored as returned.
• Five failed attempts preceded the scored success in 4 of the 180 runs; only the final attempt is scored.
• Several applications are intentionally vulnerable demonstrations, many vulnerabilities were seeded, and Java is thin; how the results carry over to production code bases is untested.

09 · Reproduction

Trace the release to its public archive

All analysis numbers were recomputed from the public run archive by a separate re-implementation of the scorer, which reproduces every recorded scoring decision in all 180 runs.

The main dataset is publications/vulnbench-v2/ in the harness repository, with a SHA-256 checksum for the archive and per-file checksums.

Open data and provenance

Terms

Plain-language terms

Attacker-reachable vulnerability

A flaw where input an outside attacker controls reaches a dangerous operation without being neutralized.

Reference set

The 108 curated vulnerabilities, each with its class, source, sink and flow. Results are scored against the frozen snapshot stored in the run manifest; the public findings-attacker-reachable.json files match it except for iguana, which now carries a correction.

Source

Where attacker-controlled input enters the application.

Sink

The dangerous operation the input reaches.

Cross-file flow

A curated flow whose source-to-sink path spans more than one file.

Endpoint rule

The match rule: the right class, and reported locations within two lines of the curated flow's endpoints; for flows of three or more locations, both a source and a sink.

Endpoint-aware configuration

One of the seven configurations scored with the endpoint rule. DeepSec, which reports no endpoint roles, is not one.

Configuration

A complete harness and model combination, such as Claude Code /security-review with Opus 5 at xhigh effort.

Harness

The product around the model: its built-in instructions, pipeline and reporting.

Matched

Credited to a reference vulnerability under the endpoint rule.

Outside the reference set

A finding that matched no reference vulnerability. It is not necessarily wrong.

Macro recall

Recall computed per application and averaged, so each application weighs the same.

Localized recall

DeepSec's metric: class match plus any reported location near any curated flow location.

Classes the built-in review policy excludes or restricts

ReDoS, prototype pollution and open redirect, chosen from Claude Code's /security-review prompt text.

Prompt-swap control

A supplementary run of the same model and effort with the benchmark's default audit prompt in place of /security-review.