Test Readability Optimization in Context of Symbolic Execution
https://doi.org/10.15514/ISPRAS-2016-28(5)-14
Abstract
Software testing is a time consuming process. In general, software companies spend about 50% of development time on testing. On the other hand, lack of testing implies financial and other risks. In the case of systematic testing a suitable test suite should provide an appropriate code coverage. A lot of code-based test generators have been developed in order to provide systematic code coverage. Such tools tend to produce lots of almost unreadable test suites which hard to verify manually. This problem is formulated as a Human Oracle Cost Problem. This work introduces a method for readability optimization of automatically generated test suites in context of Symbolic Execution. It uses natural language model in order to optimze each test case. This conception has been applied first in the Search Based Testing paradyghm. In contrast of existing search based tool proposed DSE-based tool uses SMT-solver in order to incrementally improve readability of a single test for a single program path. To validate proposed method a tool on top of LLVM and CVC4 frameworks is created. Experimental evaluation on 12 string processing routins from the Linux repository shows that this method can improve test inputs gracefully.
About the Authors
I. A. Yakimov
Institute of space and informatic technologies, Siberian Federal University
Russian Federation
Russian Federation
A. S. Kuznetsov
Institute of space and informatic technologies, Siberian Federal University
Russian Federation
Russian Federation
References
1. Hambling B. Realistic and Cost-effective Software Testing. Kelly, Management and Measurement of Software Quality, UNICOM SEMINARS, Middlesex, UK, 1993, pp. 95-112.
2. Tassey G. The economic impacts of inadequate infrastructure for software testing. National Institute of Standards and Technology, Final Report, May 2002.
3. Saswat A., Burke E.K., Chen T.Y., Clark J., Cohen M.B., Griskamp W., Harman M., Harrold M.J., McMinn P. An orchestrated survey of methodologies for automated software test case generation. Journal of Systems and Software, 2013, vol. 86, issue 8, pp. 1978-2001.
4. Cadar C., Godefroid P., Khundish S., Păsăreanu C.S., Sen Koushik, Tillman N., Visser W. Symbolic Execution for Software Testing in Practice – Preliminary Assessment. ICSE'11 Proceedings of the 33rd International Conference on Software Engineering, 2011, pp. 1066-1071.
5. Barr E.T., Harman M., Shahbaz M., Yoo S. The Oracle Problem in Software Testing: A Survey. IEEE Transactions on Software Engineering, 2015, vol. 41, issue 5, pp. 507-525.
6. Afshan S., McMinn P., Stevenson M. Evolving readable string test inputs using a natural language model to reduce human oracle cost. International Conference on Software Testing, Verification and Validation, 2013.
7. Tracey N., Clark J., Mander K., McDermid J. An automated framework for structural test-data generation. Proceedings of the International Conference on Automated Software Engineering, 1998, pp. 285–288.
8. King J.C. Symbolic execution and program testing. Communication of The ACM, 1976, vol. 19, issue 17, pp. 385-394.
9. Cadar C., Ganesh V., Pawlowski P.M., Dill D.L., Engler D.R. EXE: automatically generating inputs of death. Proceedings of the 13th ACM conference on Computer and Communication security, 2006, pp 322-335.
10. Sen K., Marinov D., Agha G.. CUTE: a concolic unit testing engine for C. ESEC/FSE-13 Proceedings of the 10th European software engineering conference held jointly with 13th ACM SIGSOFT international symposium on Foundationsof software engineering, 2006, vol. 30, issue 5, pp. 263-272
11. J. H. Holland. Adaptation in Natural and Artificial Systems. University of Michigan Press, Ann Arbor, 1975.
12. LLVM: http://llvm.org/
13. CVC4: http://cvc4.cs.nyu.edu/web/
14. Jones M. N., Mewhort D.J.K. Case-sensitive letter and bigram frequency counts from large-scale English corpora, 2004, vol. 36, issue 3, pp. 388
15. Linux: https://github.com/torvalds/linux/tree/master/kernel
16. The 100 commonest English words https://en.oxforddictionaries.com/explore/what-can-corpus-tell-us-about-language
Review
For citations:
Yakimov I.A., Kuznetsov A.S. Test Readability Optimization in Context of Symbolic Execution. Proceedings of the Institute for System Programming of the RAS (Proceedings of ISP RAS). 2016;28(5):227-238. (In Russ.) https://doi.org/10.15514/ISPRAS-2016-28(5)-14
Email this article 