Citation :

Naga Raju Doddi, "Same Code, Different Findings: An Empirical Study of Run-to-Run Instability in Rule-Based and LLM-Based Security Scanners," International Journal of Computer Trends and Technology (IJCTT), vol. 74, no. 9, pp. 44-59, 2026. Crossref, https://doi.org/10.14445/22312803/IJCTT-V74I9P105

Abstract

Security scanners in CI/CD pipelines are believed to give the same answers for the same code. When a finding disappears, it is treated as fixed, and when a scan stage finds no blocking vulnerabilities, the build proceeds to production. This study tests that assumption. It ran 350 rule-based scans of 12 projects with Opengrep and CodeQL under normal, low-memory and busy-CPU conditions, and scanned 172 labelled cases five times each with four LLMs. For the rule-based scanners, resource budgets silently change the results: in one project, 32% of the findings appear and disappear across runs of the same commit ID. This study shows that when LLM-based scanners are added as a build step in CI/CD, for 3 out of the 4 models, between 28.2% and 91.1% of the findings do not occur across same runs; setting the temperature to 0 reduces but doesn't remove the variation, and new models are no longer accepting a temperature setting. Unstable LLM findings are more likely to be off-target noise, even on novel code. One scan recovers only 39–98% of the findings of five scans, and keeping only findings reported in all five runs raises the share of true positives at almost no loss of recall. Inconsistency means opposite things for the two types of scanners: a lost detection for the rule-based engines and noise for the LLMs. These results call for CI build gates that treat incomplete scans as failures, and for remediation tracking that does not treat a missing finding as a fix.

Keywords

Continuous integration, DevSecOps, Large language models, Nondeterminism, Static application security testing.

References

[1] Snyk, Troubleshooting Snyk Reports. [Online]. Available: https://docs.snyk.io/scan-fix-and-prevent/prevent/analytics/reports-tab/troubleshooting-snyk-reports

[2] Checkmarx, SAST Scanner. [Online]. Available: https://docs.checkmarx.com/en/34965-324470-sast-scanner.html

[3] Brittany Johnson et al., “Why don't Software Developers use Static Analysis Tools to find Bugs?,” 2013 35^th International Conference on Software Engineering (ICSE), San Francisco, CA, USA, 2013.
[CrossRef] [Google Scholar] [Publisher Link]

[4] Maria Christakis, and Christian Bird, “What Developers Want and Need from Program Analysis: An Empirical Study,” Proceedings of the 31^st IEEE/ACM International Conference on Automated Software Engineering (ASE), Singapore, pp. 332-343, 2016.
[CrossRef] [Google Scholar] [Publisher Link]

[5] Shuyin Ouyang et al., “An Empirical Study of the Non-determinism of ChatGPT in Code Generation,” ACM Transactions on Software Engineering and Methodology, vol. 34, no. 2, pp.  1-28, 2025.
[CrossRef] [Google Scholar] [Publisher Link]

[6] Liran Tal, Snyk VulnBench JS 1.0: Can LLMs Find the same Bug Twice?, Snyk, 2026. [Online]. Available: https://snyk.io/blog/snyk-vulnbench-js-1-0-llm-security-review-repeatability/

[7] Stephen Lipp, Sebastian Banescu, and Alexander Pretschner, “An Empirical Study on the Effectiveness of Static C Code Analyzers for Vulnerability Detection,” Proceedings of the 31^st ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), Virtual, South Korea, pp. 544-555, 2022.
[CrossRef] [Google Scholar] [Publisher Link]

[8] Kaixuan Li et al., “Comparison and Evaluation on Static Application Security Testing (SAST) Tools for Java,” Proceedings of the 31^st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), San Francisco, CA, USA, pp. 921-933, 2023.
[CrossRef] [Google Scholar] [Publisher Link]

[9] Berk Atıl et al., “Non-determinism of “Deterministic” LLM System Settings in Hosted Environments,” Proceedings of the 5^th Workshop on Evaluation and Comparison of NLP Systems, Mumbai, India, pp. 135-148, 2025.
[CrossRef] [Google Scholar] [Publisher Link]

[10] Saad Ullah et al., “LLMs cannot Reliably Identify and Reason about Security Vulnerabilities (Yet?): A Comprehensive Evaluation, Framework, and Benchmarks,” 2024 IEEE Symposium on Security and Privacy (SP), San Francisco, CA, USA, pp. 862-880, 2024.
[CrossRef] [Google Scholar] [Publisher Link]

[11] Yangruibo Ding et al., “Vulnerability Detection with Code Language Models: How Far are we?,” 2025 IEEE/ACM 47^th International Conference on Software Engineering (ICSE), Ottawa, ON, Canada, pp. 1729-1741, 2025.
[CrossRef] [Google Scholar] [Publisher Link]

[12] Avishree Khare et al., “Understanding the Effectiveness of Large Language Models in Detecting Security Vulnerabilities,” 2025 IEEE Conference on Software Testing, Verification and Validation (ICST), Napoli, Italy, pp. 103-114, 2025.
[CrossRef] [Google Scholar] [Publisher Link]

[13] OWASP, OWASP Benchmark. [Online]. Available: https://owasp.org/projects/benchmark

[14] Roshan N. Rajapakse et al., “Challenges and Solutions when Adopting DevSecOps: A Systematic Review,” Information and Software Technology, vol. 141, pp. 1-26, 2022.
[CrossRef] [Google Scholar] [Publisher Link]

[15] Opengrep. [Online]. Available: https://github.com/opengrep/opengrep

[16] GitHub, Codeql-cli-binaries. [Online]. Available: https://github.com/github/codeql-cli-binaries

[17] Ziyang Li, Saikat Dutta, and Mayur Naik, “IRIS: LLM-assisted Static Analysis for Detecting Security Vulnerabilities,” International Conference on Learning Representations (ICLR), vol. 2025, pp. 1-24, 2025.
[Google Scholar] [Publisher Link]

[18] Nasif Imtiaz, Seaver Thorn, and Laurie Williams, “A Comparative Study of Vulnerability Reporting by Software Composition Analysis Tools,” Proceedings of the 15^th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), pp. 1-11, 2021.
[CrossRef] [Google Scholar] [Publisher Link]

[19] Huimin Hu et al., “An Empirical Study of Suppressed Static Analysis Warnings,” Proceedings of the ACM Software Engineering (FSE), pp. 290-311, 2025.
[CrossRef] [Google Scholar] [Publisher Link]

[20] Larissa Braz et al., “An Exploratory Study on Regression Vulnerabilities,” Proceedings of the 16^th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), pp. 12-22, 2022.
[CrossRef] [Google Scholar] [Publisher Link]

[21] Sarah Heckman, and Laurie Williams, “A Systematic Literature Review of Actionable Alert Identification Techniques for Automated Static Code Analysis,” Information and Software Technology, vol. 53, no. 4, pp. 363-387, 2011.
[CrossRef] [Google Scholar] [Publisher Link]

[22] Zhaoqiang Guo et al., “Mitigating False Positive Static Analysis Warnings: Progress, Challenges, and Opportunities,” IEEE Transactions on Software Engineering, vol. 49, no. 12, pp. 5154-5188, 2023.
[CrossRef] [Google Scholar] [Publisher Link]

[23] Xiuting Ge et al., “Machine Learning for Actionable Warning Identification: A Comprehensive Survey,” ACM Computing Surveys, vol. 57, no. 2, pp. 1-35, 2024.
[CrossRef] [Google Scholar] [Publisher Link]

[24] Anant Kharkar et al., “Learning to Reduce False Positives in Analytic Bug Detectors,” Proceedings of the 44^th International Conference on Software Engineering (ICSE), pp. 1307-1316, 2022.
[CrossRef] [Google Scholar] [Publisher Link]

[25] Hong Jin Kang, Khai Loong Aw, and David Lo, “Detecting False Alarms from Automatic Static Analysis Tools: How far are we?,” Proceedings of the 44^th International Conference on Software Engineering (ICSE), pp. 698-709, 2022.
[CrossRef] [Google Scholar] [Publisher Link]

[26] Jaime Spacco, David Hovemeyer, and William Pugh, “Tracking Defect Warnings Across Versions,” Proceedings of the 2006 International Workshop on Mining Software Repositories (MSR), Shanghai, China, pp. 133-136, 2006.
[CrossRef] [Google Scholar] [Publisher Link]

[27] Pavel Avgustinov et al., “Tracking Static Analysis Violations over Time to Capture Developer Characteristics,” 2015 IEEE/ACM 37^th IEEE International Conference on Software Engineering (ICSE), Florence, Italy, pp. 437-447, 2015.
[CrossRef] [Google Scholar] [Publisher Link]

[28] Miao Miao et al., “An Extensive Empirical Study of Nondeterministic Behavior in Static Analysis Tools,” 2025 IEEE/ACM 47^th International Conference on Software Engineering (ICSE), Ottawa, Canada, pp. 1064-1076, 2025.
[CrossRef] [Google Scholar] [Publisher Link]

[29] Qingzhou Luo et al., “An Empirical Analysis of Flaky Tests,” Proceedings of the 22^nd ACM SIGSOFT International Symposium Foundations of Software Engineering (FSE), pp. 643-653, 2014.
[CrossRef] [Google Scholar] [Publisher Link]

[30] Owain Parry et al., “A Survey of Flaky Tests,” ACM Transactions on Software Engineering and Methodology, vol. 31, no. 1, pp. 1-74, 2022.
[CrossRef] [Google Scholar] [Publisher Link]

[31] Austin Mordahl et al., “ECSTATIC: An Extensible Framework for Testing and Debugging Configurable Static Analysis,” 2023 IEEE/ACM 45^th International Conference on Software Engineering (ICSE), Melbourne, Australia, pp. 550-562, 2023.
[CrossRef] [Google Scholar] [Publisher Link]

[32] Cheng Wen et al., “Automatically Inspecting Thousands of Static Bug Warnings with Large Language Model: How Far are we?,” ACM Transactions on Knowledge Discovery from Data, vol. 18, no. 7, pp. 1-34, 2024.
[CrossRef] [Google Scholar] [Publisher Link]

[33] Binyuan Hui et al., “Qwen2.5-Coder Technical Report,” arXiv preprint, pp. 1-32, 2024.
[CrossRef] [Google Scholar] [Publisher Link]

[34] Ollama. [Online]. Available: https://github.com/ollama/ollama

[35] Claude Platform Docs, Models Overview. [Online]. Available: https://platform.claude.com/docs/en/models/overview