TSAR: Tool for Static Analyzers Ranking

The article presents a new tool, TSAR, designed for evaluating the effectiveness of static analyzers. TSAR includes three main components: a static analyzer assessment system, a test generator based on the Common Weakness Enumeration (CWE), and code transformation mechanisms (mutators) to challenge the analyzers. The assessment system identifies weaknesses in static analysis tools, while the test generator creates specific cases based on known vulnerabilities. Code transformations create complex structures that complicate analysis and intended to test the analyzers' ability in detecting real vulnerabilities. This tool provides researchers and developers with an opportunity for a deeper assessment of the quality of software static analyzers for their further improvement.

1. CWE, Common Weakness Enumeration, https://cwe.mitre.org, accessed 01.12.2024.

2. Cuoq, P. et al. (2012). Testing Static Analyzers with Randomly Generated Programs. In: Goodloe, A.E., Person, S. (eds) NASA Formal Methods. NFM 2012. Lecture Notes in Computer Science, vol 7226. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-28891-3_12

3. Yang X. et al. Finding and understanding bugs in C compilers // Proceedings of the 32nd ACM SIGPLAN conference on Programming language design and implementation. – 2011. – С. 283-294.

4. Arusoaie A., Ciobâca S., Craciun V., Gavrilut D. and Lucanu D., "A Comparison of Open-Source Static Analysis Tools for Vulnerability Detection in C/C++ Code," 19th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC), Timisoara, Romania, 2017, pp. 161-168, doi: 10.1109/SYNASC.2017.00035.

5. Marjamaki D., “Cppcheck - a tool for static c/c++ code analysis”, http://cppcheck.wiki.sourceforge.net/

6. Calcagno C. et al. Moving fast with software verification // NASA Formal Methods Symposium. – Cham : Springer International Publishing, 2015. – С. 3-11.

7. Clang, “Clang Static Analyzer”, https://clang-analyzer.llvm.org/

8. OCLint - A static source code analysis tool to improve quality and reduce defects for C, C++ and Objective-C, https://github.com/oclint/oclint, accessed 01.12.2024.

9. Shiraishi S., Mohan V., Marimuthu H. Test suites for benchmarks of static analysis tools // 2015 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW). – IEEE, 2015. – С. 12-15.

10. Christian Klinger, Maria Christakis, and Valentin Wüstholz. 2019. Differentially testing soundness and precision of program analyzers. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2019). Association for Computing Machinery, New York, NY, USA, 239–250. https://doi.org/10.1145/3293882.3330553

11. Groce A. et al., "Evaluating and Improving Static Analysis Tools Via Differential Mutation Analysis," 2021 IEEE 21st International Conference on Software Quality, Reliability and Security (QRS), Hainan, China, 2021, pp. 207-218, doi: 10.1109/QRS54544.2021.00032.

12. Stephan Lipp, Sebastian Banescu, and Alexander Pretschner. 2022. An empirical study on the effectiveness of static C code analyzers for vulnerability detection. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2022). Association for Computing Machinery, New York, NY, USA, 544–555

13. The OpenSSF CVE Benchmark, https://github.com/ossf-cve-benchmark/ossf-cve-benchmark, accessed 01.12.2024.

14. Boland T., Black P. E. Juliet 1.1 C/C++ and java test suite // Computer. – 2012. – Т. 45. №. 10. С. 88-90.

15. saveourtool: Test framework for Static Analyzers and Compilers, https://github.com/saveourtool, accessed 01.12.2024.

16. ГОСТ Р 71207-2024 – Защита информации. Разработка безопасного программного обеспечения. Статический анализ программного обеспечения. Общие требования. https://protect.gost.ru/document1.aspx?control=31&baseC=6&page=3&month=2&year=2024&search=&id=257752

17. Lattner C. et al. MLIR: A compiler infrastructure for the end of Moore's law // arXiv preprint arXiv:2002.11054. – 2020.

18. Static Analysis Results Interchange Format (SARIF),

19. https://docs.oasis-open.org/sarif/sarif/v2.1.0/sarif-v2.1.0.html, accessed 01.12.2024.

20. 'emitc' Dialect to generate C/C++ from MLIR, https://mlir.llvm.org/docs/Dialects/EmitC/, accessed 01.12.2024.

21. Banescu S., Pretschner A. A tutorial on software obfuscation // Advances in Computers. – 2018. – Т. 108. – С. 283-353.

22. Jing D. Improvement of Vulnerable Code Dataset Based on Program Equivalence Transformation // Journal of Physics: Conference Series. – IOP Publishing, 2022. – Т. 2363. – №. 1. – С. 012010.

23. Li Y. et al. A closer look into transformer-based code intelligence through code transformation: Challenges and opportunities // arXiv preprint arXiv:2207.04285. – 2022.

24. Clang Tidy. https://clang.llvm.org/extra/clang-tidy/, accessed 01.12.2024.

25. Charoenwet W. et al. An empirical study of static analysis tools for secure code review // Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. – 2024. – С. 691 - 703.

26. Treat C++ 'throw' as a sink jrose-apple // committed on Aug 18, 2012 https://github.com/llvm/llvm-project/commit/a4309c941c622f45f5fe58faaa0227a7f8b4da16, accessed 01.12.2024.