VulnBench
Menu

About VulnBench

A durable home for inspectable benchmark evidence

Snyk VulnBench publishes versioned studies of vulnerability-finding behavior, their methods, datasets, cases, and interpretation boundaries.

Product promise

Explore the result. Inspect the evidence.

Explore how reliably AI systems find vulnerabilities—and inspect the evidence behind every conclusion.

VulnBench is not a universal leaderboard, a live scanner, or the home for all Snyk AI security research. It is one benchmark initiative spanning versioned protocols and stable public releases.

Initiative principles

Scientific framing stays in the interface

  1. Evidence before ranking. Lead with measured behavior.
  2. Precise language. Reference agreement is not renamed accuracy.
  3. Every value has provenance. Release, dataset, metric, aggregation, and sample size travel together.
  4. Version boundaries are explicit. Incompatible protocols do not produce false comparisons.
  5. Immutable releases. Corrections are documented rather than silently rewritten.

Team

Researchers and authors

Snyk VulnBench 2.0

  • Liran TalSnyk
  • Chris LiuSnyk
  • Charlotte FeredaySnyk
  • Sebastian RothSnyk
  • Brian ClarkSnyk

Snyk VulnBench JS 1.0

  • Liran TalSnyk
  • Johannes KloosSnyk
  • Arsenii RudichSnyk
  • Stephen ThoemmesSnyk
  • Manoj NairSnyk

Snyk VulnBench 2.0

Disclosure

Citation guidance

Cite the release, not an unversioned score

Snyk VulnBench 2.0

Liran Tal, Chris Liu, Charlotte Fereday, Sebastian Roth, Brian Clark. “How Deterministic SAST and AI Security Agents Divide the Work: Coverage, Harness Policy and Cost in Snyk VulnBench 2.0.” arXiv:2610.38617, 2026.
@misc{tal2026snykvulnbench20,
  author = {Tal, Liran and Liu, Chris and Fereday, Charlotte and Roth, Sebastian and Clark, Brian},
  title = {How Deterministic SAST and AI Security Agents Divide the Work: Coverage, Harness Policy and Cost in Snyk VulnBench 2.0},
  year = {2026},
  eprint = {2610.38617},
  archivePrefix = {arXiv},
  primaryClass = {cs.CR},
  doi = {10.48550/arXiv.2610.38617},
  url = {https://arxiv.org/abs/2610.38617}
}

Liran Tal · Chris Liu · Charlotte Fereday · Sebastian Roth · Brian Clark

Include the release name and dataset version 2.0.0 when reproducing a chart or filtered result.

Open the canonical paper

Snyk VulnBench JS 1.0

Liran Tal, Johannes Kloos, Arsenii Rudich, Stephen Thoemmes, Manoj Nair. “Snyk VulnBench JS 1.0: Can LLMs Find the Same Bugs Twice?.” arXiv, 2026.
@misc{tal2026snykvulnbenchjs10,
  author = {Tal, Liran and Kloos, Johannes and Rudich, Arsenii and Thoemmes, Stephen and Nair, Manoj},
  title = {Snyk VulnBench JS 1.0: Can LLMs Find the Same Bugs Twice?},
  year = {2026},
  url = {https://arxiv.org/abs/2606.15762}
}

Liran Tal · Johannes Kloos · Arsenii Rudich · Stephen Thoemmes · Manoj Nair

Include the release name and dataset version1.0.0 when reproducing a chart or filtered result.

Open the canonical paper