01 · Benchmark design
300 scans under repeated conditions
The benchmark contains ten small JavaScript and Express fixture projects with 44 Snyk Code reference findings. Six configurations ran every task five times, producing 300 total scans: 250 model runs and 50 deterministic Snyk Code runs.
Model configurations used the same direct audit prompt through a Claude Code harness and returned structured findings. They could read project files but not each fixture’s referencefindings.json.
02 · Reference matching
Agreement uses a deliberately lenient scorer
A model finding receives reference credit when it reports the same vulnerability type as a Snyk Code finding. It does not need to identify the same file, line, severity, or exact source-to-sink path.
03 · Metric definitions
Each value answers a narrow question
- Finding recurrence
The number of identical repetitions in which a normalized finding signature appeared.
Unit: runs out of 5 · Aggregation: Unique normalized finding signaturesNormalization choices affect recurrence percentages and do not establish ground-truth validity.- Snyk-reference F1
The harmonic mean of precision and recall when Snyk Code findings define the reference set.
Unit: percent · Aggregation: Macro average across benchmark projectsReference agreement is not universal vulnerability-detection accuracy.- Recall against the Snyk Code reference set
The share of Snyk Code reference findings matched by a configuration.
Unit: percent · Aggregation: Macro average across benchmark projectsThe scorer matches vulnerability type without requiring the same file, line, severity, or data-flow identity.- Precision against the Snyk Code reference set
The share of reported findings that match a Snyk Code reference finding.
Unit: percent · Aggregation: Macro average across benchmark projectsUnmatched reports can be false-positive-shaped, adjacent, or valid product-gap candidates.- Average session duration
Mean wall-clock duration for one benchmark session.
Unit: seconds · Aggregation: Mean across repeated runsSmall benchmark fixtures do not represent the latency of every production repository.- Estimated model-session cost
Estimated API-equivalent model cost under the publication's pricing assumptions.
Unit: USD · Aggregation: Mean across repeated model runsSnyk Code has no comparable model-session cost and published values reflect small fixtures.
04 · Resource projections
A release-wide linear extrapolation
The Explorer labels these values Projected: they are not completed scans of the application references. The lower Efficiency figure remains Observedbenchmark evidence, including Snyk-reference F1 as reference agreement rather than universal detection accuracy.
For each configuration and resource metric, the release resource total is the sum of the ten published per-fixture mean session resource values. The resource rate per 1,000 SLOC is that total divided by 1.154; projected application resource is the rate multiplied by the application SLOC and divided by 1,000. Rates always use the complete release matrix, not the currently filtered fixture subset; filters determine only which configuration lines are visible.
- Fixture denominator
- 1,154 SLOC across the JS 1.0 fixtures, counted with
cloc 2.06across JavaScript, TypeScript, JSX/TSX, CSS/SCSS, HTML, and Handlebars. - Excluded from the denominator
- Package manifests, lockfiles, dependencies, comments, and blank lines.
- Resource semantics
- Cost uses estimated API-equivalent model-session USD; tokens use mean total tokens; duration uses mean wall-clock seconds. Snyk Code has no comparable model-session cost or token value.
Pinned web-application size references
cloc 2.06 measured production application code on the following pinned upstream revisions on 2026-08-29. The counts include browser and server application code and exclude tests, documentation, dependencies, generated output, fixtures, mocks, and build artifacts.
| Application | Pinned upstream revision | Production SLOC |
|---|---|---|
| Mastodon | b59ddc79241a9e72415d61334b3f748a916c789a | 170,480 |
| Zulip | 49ed929428a8bada600dfb2fc51432135e045396 | 326,630 |
| WordPress Core | d30211ee316ed5d41d3d1531e395440fa9f2ce6b | 476,469 |
| Discourse | 768a4ed1cd8e6742fe1c1340a9c4ab01318285ec | 637,925 |
| Mattermost | c8b1cc0046c9ab53de4cb33804a7b4bc5cd03a83 | 795,858 |
05 · Limitations
What the protocol cannot establish
- Snyk Code defines the reference set; its 100% score is deterministic reproduction of that set, not universal accuracy.
- The scorer matches by vulnerability type and does not require the same file, line, severity, or source-to-sink identity.
- The fixtures are small JavaScript and Express projects and do not represent every production architecture.
- Unmatched reports may include false positives, adjacent comments, or valid product-gap candidates.
- Normalization choices affect recurrence percentages.
- Cost values reflect small fixtures and the publication's model-session assumptions.
06 · Reproduction
Trace the release to immutable source artifacts
The website vendors the exact upstream source tree at commit7c944ea438a31ea4cbd6803f1bb9560d01f932e5. Its JSONL results, fixture references, generated visualizations, and Git blob manifest are available through the data page.