01 · Benchmark design
300 scans under repeated conditions
The benchmark contains ten small JavaScript and Express fixture projects with 44 Snyk Code reference findings. Six configurations ran every task five times, producing 300 total scans: 250 model runs and 50 deterministic Snyk Code runs.
Model configurations used the same direct audit prompt through a Claude Code harness and returned structured findings. They could read project files but not each fixture’s referencefindings.json.
02 · Reference matching
Agreement uses a deliberately lenient scorer
A model finding receives reference credit when it reports the same vulnerability type as a Snyk Code finding. It does not need to identify the same file, line, severity, or exact source-to-sink path.
03 · Metric definitions
Each value answers a narrow question
- Finding recurrence
The number of identical repetitions in which a normalized finding signature appeared.
Unit: runs out of 5 · Aggregation: Unique normalized finding signaturesNormalization choices affect recurrence percentages and do not establish ground-truth validity.- Snyk-reference F1
The harmonic mean of precision and recall when Snyk Code findings define the reference set.
Unit: percent · Aggregation: Macro average across benchmark projectsReference agreement is not universal vulnerability-detection accuracy.- Recall against the Snyk Code reference set
The share of Snyk Code reference findings matched by a configuration.
Unit: percent · Aggregation: Macro average across benchmark projectsThe scorer matches vulnerability type without requiring the same file, line, severity, or data-flow identity.- Precision against the Snyk Code reference set
The share of reported findings that match a Snyk Code reference finding.
Unit: percent · Aggregation: Macro average across benchmark projectsUnmatched reports can be false-positive-shaped, adjacent, or valid product-gap candidates.- Average session duration
Mean wall-clock duration for one benchmark session.
Unit: seconds · Aggregation: Mean across repeated runsSmall benchmark fixtures do not represent the latency of every production repository.- Estimated model-session cost
Estimated API-equivalent model cost under the publication's pricing assumptions.
Unit: USD · Aggregation: Mean across repeated model runsSnyk Code has no comparable model-session cost and published values reflect small fixtures.
04 · Limitations
What the protocol cannot establish
- Snyk Code defines the reference set; its 100% score is deterministic reproduction of that set, not universal accuracy.
- The scorer matches by vulnerability type and does not require the same file, line, severity, or source-to-sink identity.
- The fixtures are small JavaScript and Express projects and do not represent every production architecture.
- Unmatched reports may include false positives, adjacent comments, or valid product-gap candidates.
- Normalization choices affect recurrence percentages.
- Cost values reflect small fixtures and the publication's model-session assumptions.
05 · Reproduction
Trace the release to immutable source artifacts
The website vendors the exact upstream source tree at commit7c944ea438a31ea4cbd6803f1bb9560d01f932e5. Its JSONL results, fixture references, generated visualizations, and Git blob manifest are available through the data page.