Testing of Operating Systems
https://doi.org/10.15514/ISPRAS-2014-26(1)-3
Abstract
An operating system is a base stone of any computer system. Failures and bugs in operating system impact the functionality of the system as a whole, that is why correctness and reliability of operating systems are so important. A variety of circumstances make verification and testing of operating systems a complicated issue. The list includes high dependence of operating systems on hardware, their massive internal concurrency, huge number of configuration options, required tolerance to aggressive actions of counteragents and hardware faults, a need for long continuous work without reboot, etc. The paper discusses influence of all the circumstances on testing, describes testing tools and techniques developed in ISPRAS and presents our experience of testing of various components of Linux as well as a few other real-time operating systems.
About the Authors
E. A. Gerlits
Institute for System Programming of RAS
Russian Federation
Russian Federation
V. V. Kuliamin
Institute for System Programming of RAS
Russian Federation
Russian Federation
A. V. Maksimov
Institute for System Programming of RAS
Russian Federation
Russian Federation
A. K. Petrenko
Institute for System Programming of RAS
Russian Federation
Russian Federation
A. V. Khoroshilov
Institute for System Programming of RAS
Russian Federation
Russian Federation
A. V. Tsyvarev
Institute for System Programming of RAS
Russian Federation
Russian Federation
References
1. V.V. Kulyamin, N.V. Pakulin, O.L. Petrenko, А.А. Sortov, А.V. KHoroshilov. Formalizatsiya trebovanij na praktike [Requirements formalization in practice]. Preprinty ISP RАN [The preprints of ISP RAS], 2006, no. 13, pp. 1-70 (in Russian).
2. V.V. Kulyamin, А.K. Petrenko, V.V. Rubanov, А.V. KHoroshilov. Formalizatsiya interfejsnykh standartov i avtomaticheskoe postroenie testov sootvetstviya [Formalization of interface standards and automatic construction of conformance test suites]. Informatsionnye tekhnologii [Information technologies], 2007, vol. 8, pp. 1-8 (in Russian).
3. M.V. Ekimov, I.V. Koverninskij, А.V. KHoroshilov. АRM PT: sozdanie sistemy proektirovaniya testov dlya kriticheskikh sistem na osnove SPO [Automated Working Place of Test Developer: developing a test design system for safety critical systems using free software]. Tezisy dokladov Vserossijskoj konferentsii “Svobodnoe programmnoe obespechenie” [The theses of All-Russian conferene “Free software – 2010”], 2010, pp. 64-65 (in Russian).
4. Requality tool, http://forge.ispras.ru/projects/reqdb.
5. D. Yu. Kichigin. A method of test-suite reduction for regression integration testing. Programming and Computer Software, 2009, vol. 35, no. 5, pp. 57-69. doi: 10.1134/S0361768809050041.
6. Alexey Khoroshilov, Vladimir Rubanov, Eugene Shatokhin. Automated Formal Testing of C API Using T2C Framework. The Proceedings of the Third International Symposium «Leveraging Applications of Formal Methods, Verification and Validation» (ISoLA 2008), 2008, pp. 56-70. doi: 10.1007/978-3-540-88479-8_5.
7. TETWare User Guide, http://tetworks.opengroup.org/documents/3.7/uguide.pdf.
8. ISO/IEC 23360-1-8:2005, Linux Standard Base (LSB) Core Specification 3.1. Geneve: ISO, 2005.
9. А.V. KHoroshilov. Linux Standard Base: istoriya uspekha? [Linux Standard Base: a success story?], Trudy ISP RАN [The Proceedings of ISP RAS], 2006, vol. 10, pp. 29-50 (in Russian).
10. List of errors found during testing of LINUX libraries, http://linuxtesting.org/results/impl_reports.
11. N.V. Pakulin, A.V. Khoroshilov. Development of formal models and conformance testing for systems with asynchronous interfaces and telecommunications protocols. Programming and Computer Software, 2007, vol. 33, no. 6, pp. 316-335.
12. А.V. Barantsev, I.B. Burdonov, А.V. Demakov, S.V. Zelenov, А.S. Kosachev, V.V. Kulyamin, V.А. Omel'chenko, N.V. Pakulin, А.K. Petrenko, А.V. KHoroshilov. Podkhod UniTesK k razrabotke testov: dostizheniya i perspektivy [UniTESK approach to test development: achievements and prospects]. Trudy ISP RАN [The Proceedings of ISP RAS], 2004, vol. 5, pp. 121-156 (in Russian).
13. А.V. KHoroshilov. Spetsifikatsiya i testirovanie sistem s asinkhronnym interfejsom [Specification and testing of systems with asynchronous interface]. Preprinty ISP RАN [The preprints of ISP RAS], 2006, no. 12, pp. 1-140 (in Russian).
14. IEEE 1003.1-2008, Information Technology — Portable Operating System Interface (POSIX). New York: IEEE, 2008.
15. ARINC Specification 653P1-3: Avionics Application Software Standard Interface Part 1 - Required Services. Aeronautical Radio INC, Maryland, USA, 2010.
16. Alexey Grinevich, Alexey Khoroshilov, Victor Kuliamin, Denis Markovtsev, Alexander Petrenko, Vladimir Rubanov. Formal Methods in Industrial Software Standards Enforcement. The proceedings of the 6th international Andrei Ershov memorial conference on Perspectives of systems informatics, 2006, vol. 4378 of LNCS, pp. 446-455. doi: 10.1007/978-3-540-70881-0_41.
17. A. Maksimov. Requirements-based conformance testing of ARINC 653 real-time operating systems. The Proceedings of the Data Systems In Aerospace conference (DASIA 2010), 2010.
18. Fault Injection technique, https://www.kernel.org/doc/Documentation/fault-injection/fault-injection.txt.
19. Eugene Shatokhin. Using Dynamic Analysis To Hunt Down Problems in Kernel Modules. Presentation at LinuxCon Europe 2011, Czech Republic, Prague, 26-28 October 2011.
20. KEDR framework, http://linuxtesting.org/kedr.
21. Linux File System Verification (Spruce) project, http://linuxtesting.org/spruce.
22. V. Kuliamin. Standardization and Testing of Mathematical Functions. The Proceedings of 9th international Andrei Ershov memorial conference on Perspectives of systems informatics, 2009, vol. 5947 of LNCS, pp. 257-268. doi: 10.1007/978-3-642-11486-1_22.
23. R.S. Zybin, Victor V. Kuliamin, Andrey V. Ponomarenko, Vladimir V. Rubanov, E.S. Chernov. Automation of broad sanity test generation. Programming and Computer Software, 2008, vol. 34, no. 6, pp. 351-363. doi: 10.1134/S0361768808060066.
24. А.V. KHoroshilov. Аvtomaticheskaya generatsiya testov dlya C/C++ bibliotek [Automatic test suites generation for C/C++ libraries]. Tezisy dokladov Sed'moj konferentsii razrabotchikov svobodnykh programm [The theses of 7th conference of free software developers], 2010, pp. 75-78 (in Russian).
25. API Sanity Autotest tool, http://forge.ispras.ru/projects/api-sanity-autotest.
26. LSB libchk tool, http://bzr.linuxfoundation.org/loggerhead/lsb/devel/misc-test/files.
27. Android Compatibility Test Suite, http://source.android.com/compatibility.
28. chkshlib, http://osr507doc.sco.com/en/man/html.CP/chkshlib.CP.html.
29. cmpdylib, http://www.opensource.apple.com/source/cctools/cctools-795/man/cmpdylib.1.
30. cmpshlib, sys-admin.net/ebooks/unix3/mac/ch07_01.htm.
31. dpkg-gensymbols, http://man.he.net/man1/dpkg-gensymbols.
32. A. Ponomarenko, V. Rubanov, A. Khoroshilov. Sistema analiza obratnoj binarnoj sovmestimosti bibliotek Linux [System for backward compatibility analysis of LINUX libraries]. The Proceedings of International conference “Software Engineering Conference (Russia)”, SEC(R)-2009, 2009, pp. 25-31 (in Russian).
33. V.S. Mutilin, E.M. Novikov, А.V. KHoroshilov. Аnaliz tipovykh oshibok v drajverakh operatsionnoj sistemy Linux [Analysis of typical errors in LINUX drivers]. Trudy ISP RАN [The Proceedings of ISP RAS], 2012, vol. 22, pp. 349-374. DOI: 10.15514/ISPRAS-2012-22-19. (in Russian).
34. Valgrind tool, valgrind.org.
35. Kernel Memory Leak Detector, https://www.kernel.org/doc/Documentation/kmemleak.txt.
36. Helgrind tool, http://valgrind.org/docs/manual/hg-manual.html.
37. Konstantin Serebryany, Timur Iskhodzhanov. ThreadSanitizer: data race detection in practice. The Proceedings of the Workshop on Binary Instrumentation and Applications (WBIA '09), 2009, pp.62-71. doi: 10.1145/1791194.1791203.
38. Kernel Strider project, https://code.google.com/p/kernel-strider/.
39. Race Hound project, http://forge.ispras.ru/projects/race-hound.
40. Trinity project, http://codemonkey.org.uk/projects/trinity/.
41. I.K. Isaev, D.V. Sidorov, А.YU. Gerasimov, M.K. Ermakov. Avalanche: Primenenie dinamicheskogo analiza dlya avtomaticheskogo obnaruzheniya oshibok v programmakh ispol'zuyushhikh setevye sokety [Avalanche: Dynamic analysis for automatic error detection in programs working with network sockets]. The Proceedings of Real-Time Conference, 2007 15th IEEE-NPSS, 2011, vol. 21, pp. 55-70 (in Russian).
42. Vitaly Chipounov, Volodymyr Kuznetsov, and George Candea. The S2E Platform: Design, Implementation, and Applications. ACM Transactions on Computer Systems, 2012, vol. 30, issue 1, article 2, pp.1-49. doi: 10.1145/2110356.2110358.
43. A. Barbalace, A. Lunchetta, G. Manduchi, M. Moro, A. Soppelsa and C. Taliercio. Performance Comparison of VxWorks, Linux, RTAI and XENOMAI in a Hard Real-time Application. The Proceedings of Real-Time Conference, 2007 15th IEEE-NPSS, 2007, pp. 1-5. doi: 10.1109/RTC.2007.4382787.
44. M. Franke. A Quantitative Comparison of Realtime Linux Solutions. Chemnitz University of Technology, 2007.
45. M. D. Marieska, A. I. Kistijantoro and M. Subair. Analysis and Benchmarking Performance of Real Time Patch Linux and Xenomai in Serving a Real Time Application. The Proceedings of International conference on Electrical Engineering and Informatics, 2011, pp. 1-6.
46. J. H. Koh and B. W. Choi. Performance Evaluation of Real-time Mechanisms for Real-time Embedded Linux. Journal of Institute of Control, Robotics and Systems, 2012, vol. 18, no. 4, pp. 337-342 (in Korean). doi: 10.5302/J.ICROS.2012.18.4.337.
47. J. H. Koh and B. W. Choi. Real-time Performance of Real-time Mechanisms for RTAI and Xenomai in Various Running Conditions. International Journal of Control and Automation, 2013, vol. 6, no. 1, pp. 235-246.
48. V.V. Kulyamin. Kombinatornaya generatsiya programmnykh konfiguratsij OS [Combinatorial generation of software-based OS configurations]. Trudy ISP RАN [The Proceedings of ISP RAS], 2012, vol. 23, pp. 359-370. DOI: 10.15514/ISPRAS-2012-23-20. (in Russian).
49. Vladimir Rubanov, Denis Silakov. Certification Infrastructure for the Linux Standard Base (LSB). The Proceedings of the second International Workshop on Foundations and Techniques for Open Source Software Certification (OpenCert 2008), 2008. pp. 79-88.
Review
For citations:
Gerlits E.A., Kuliamin V.V., Maksimov A.V., Petrenko A.K., Khoroshilov A.V., Tsyvarev A.V. Testing of Operating Systems. Proceedings of the Institute for System Programming of the RAS (Proceedings of ISP RAS). 2014;26(1):73-108. (In Russ.) https://doi.org/10.15514/ISPRAS-2014-26(1)-3
Email this article 