Research question
Reliability starts with recurrence
If the same agent sees the same vulnerable code twice, does it report the same security issues twice?
JS 1.0 measures repeated behavior under controlled conditions. Ten inspectable JavaScript projects were reviewed by six configurations, five times each, using unchanged code, prompts, and harness settings.
Snyk Code supplies the deterministic reference set. The resulting Snyk-reference metrics describe agreement with that set; they do not claim exhaustive vulnerability-detection accuracy.
Central result
Matched findings were stable. Extra reports varied.
The recurrence distribution shows a clear contrast between findings that aligned with the Snyk Code reference set and reports outside it.
Unmatched findings can include false-positive-shaped reports, adjacent comments, or valid product-gap candidates. They require inspection.
- 134 of 158Reference-matched findings seen in all five runsInspect
- 80 of 161Unmatched findings seen in only one runInspect
- 22 of 161Unmatched findings seen in all five runsInspect
134 of 158 · Reference-matched findings seen in all five runs
View exact recurrence values
| Finding group | Count | Share |
|---|---|---|
| Reference-matched findings seen in all five runs | 134 of 158 | 84.8% |
| Unmatched findings seen in only one run | 80 of 161 | 49.7% |
| Unmatched findings seen in all five runs | 22 of 161 | 13.7% |
Complementarity
Different systems exposed different strengths
Agentic LLM review consistently recognized familiar, high-signal exploit shapes and surfaced one likely Snyk Code product gap. Deterministic SAST was stronger at systematically enumerating repeated data-flow sinks.
The practical implication is not replacement. The evidence supports combining reasoning-oriented review with deterministic coverage, then triaging where their reports diverge.
Efficiency
More expensive sessions did not automatically agree more
The model-only cost-quality comparison did not produce a simple spend-more, score-higher pattern. Cost, token usage, duration, agreement, and recurrence therefore remain separate measures.
Published cost estimates reflect small fixtures and the model-session assumptions documented for this release.
Inspect efficiency evidenceWhat this does not prove
Keep the protocol boundary visible
- Snyk Code defines the reference set; its 100% score is deterministic reproduction of that set, not universal accuracy.
- The scorer matches by vulnerability type and does not require the same file, line, severity, or source-to-sink identity.
- The fixtures are small JavaScript and Express projects and do not represent every production architecture.
- Unmatched reports may include false positives, adjacent comments, or valid product-gap candidates.
- Normalization choices affect recurrence percentages.
- Cost values reflect small fixtures and the publication's model-session assumptions.
Continue the research trail