Initiative methodology

Comparable evidence requires explicit boundaries

VulnBench releases share principles for transparent measurement and provenance while preserving protocol-specific definitions.

01
Define the protocol before reading the result
Every release declares its task, repeated conditions, reference or ground-truth model, scorer, metrics, and limitations.
02
Preserve uncertainty and recurrence
Repeated-run spread remains visible wherever an aggregate could hide variable behavior.
03
Keep values traceable
Published charts identify release, dataset version, aggregation, units, sample size, and source assets.
04
Block incompatible comparisons
Metrics only compare across releases when their definitions and protocols explicitly declare compatibility.
05
Publish corrections, never silent rewrites
Corrected data receives a new patch version and a visible record of affected values and reasons.

Current protocol

Snyk VulnBench JS 1.0

Repeatability and Snyk-reference agreement

Read the benchmark design, matching rules, metric definitions, limitations, and source-reproduction notes for the current release.