A Snyk benchmark initiative
Can LLMs find the same bugs twice?
A repeatability and Snyk-reference agreement study
We ran the same agentic security review five times against inspectable JavaScript projects to measure what recurs, what varies, and how model findings align with a deterministic Snyk Code reference set.
- 300
- scans
- 10
- projects
- 6
- configurations
- 5
- repetitions
Headline evidence · 5 identical reviews
Same review. Different results.
300 scans · 10projects · 6 configurations
- 134 of 158Reference-matched findings seen in all five runsInspect
- 22 of 161Unmatched findings seen in all five runsInspect
- 80 of 161Unmatched findings seen in only one runInspect
134 of 158 · Reference-matched findings seen in all five runs
View exact recurrence values
| Finding group | Count | Share |
|---|---|---|
| Reference-matched findings seen in all five runs | 134 of 158 | 84.8% |
| Unmatched findings seen in all five runs | 22 of 161 | 13.7% |
| Unmatched findings seen in only one run | 80 of 161 | 49.7% |
Latest evidence · JS 1.0
Same code. Same prompt. Different findings.
Across five identical runs, 84.8% of reference-matched findings recurred every time. Nearly half of unmatched reports appeared only once—evidence to inspect, not dismiss.
What the results mean in practice
An AI review is a measurement—not a verdict.
Repeat the same task and the story sharpens: many reference-matched findings hold steady; AI review and deterministic SAST reveal different blind spots; and more spend does not reliably improve Snyk-reference agreement.
- 01Open the evidence
Repeatability makes confidence visible
134 of 158 reference-matched findings appeared in every one of five identical reviews. Repetition shows which reported patterns persist instead of relying on a single run.
Interpret with care: A reference match measures agreement with Snyk Code, not independent ground-truth accuracy.
- 02Open the evidence
AI review and SAST expose different blind spots
Models surfaced high-signal exploit shapes, while deterministic Snyk Code consistently enumerated repeated data-flow sinks. Their differing results are useful evidence—not a reason to declare one a universal winner.
Interpret with care: Unmatched findings require case-level inspection before they can be classified.
- 03Open the evidence
Paying more did not reliably improve the result
In this benchmark, higher session cost did not consistently yield higher Snyk-reference F1. Spend alone is a poor shortcut for choosing a configuration.
Interpret with care: Cost estimates reflect the tested small fixtures and publication assumptions.
Benchmark anatomy
How VulnBench measures behavior
Repeat the conditions, preserve the evidence, and separate observed agreement from claims the protocol cannot support.
Read the full methodology- 1
Select inspectable projects
Ten small JavaScript and Express fixtures make every run and reference finding reviewable.
- 2
Repeat the same task
Each configuration sees the same code, prompt, harness, and task five times.
- 3
Normalize findings
Reported issues become documented signatures suitable for recurrence analysis.
- 4
Match the reference set
The scorer compares vulnerability type against deterministic Snyk Code findings.
- 5
Measure behavior
Agreement, recurrence, variance, coverage, cost, tokens, and duration stay distinct.
- 6
Inspect divergence
Unmatched reports remain evidence to investigate—not automatic false positives.
Research principles
Evidence before ranking
VulnBench is a versioned research initiative, not a universal leaderboard. Every release defines what it measured and what it did not prove.
- Transparent reference sets
- Definitions and limitations stay visible.
- Repeated measurement
- Variance is a result, not a footnote.
- Inspectable cases
- Headline claims link toward underlying evidence.
- Reproducible data
- Versioned source artifacts remain downloadable.
- Explicit limitations
- Agreement is never relabeled as accuracy.
Release history
Further releases are planned. Unpublished results will not appear as speculative rankings or empty release cards.
View release catalogPublication
Read, reproduce, and cite the work
The paper, reviewed methodology, source snapshot, and release data use stable public links.
Preferred citation
@misc{tal2026snykvulnbenchjs10,
author = {Tal, Liran and Kloos, Johannes and Rudich, Arsenii and Thoemmes, Stephen and Nair, Manoj},
title = {Snyk VulnBench JS 1.0: Can LLMs Find the Same Bugs Twice?},
year = {2026},
url = {https://arxiv.org/abs/2606.15762}
}