Automation of conformance testing for communication protocols
https://doi.org/10.15514/ISPRAS-2014-26(1)-4
Abstract
This article summarizes the experience gained while developing test suites for conformance testing of implementations of Internet protocols. The projects described in this article used the UniTESK technology as a base for constructing tests. During the development of test suites we identified certain features of the protocols that make it difficult to apply UniTESK unmodified to protocol conformance testing, as well as some limitations of UniTESK toolkits. Nevertheless all these obstacles were successfully overcome without going beyond UniTESK concepts.
About the Authors
Nikolay Pakulin
Institute for System Programming of RAS
Russian Federation
Russian Federation
Victor Shnitman
Institute for System Programming of RAS
Russian Federation
Russian Federation
Alexey Nikeshin
Institute for System Programming of RAS
Russian Federation
Russian Federation
References
1. CCITT Recommendation Z.100. Specification and Description Language (SDL). Geneve, Switzerland: ITU, 1993. 245 p.
2. ISO/IEC 9074. Information Processing Systems — Open Systems Interconnection. Estelle — A Formal Description Technique based on an Extended State Transition Model. Geneve, Switzerland: ISO, 1989 1 я редакция,1997 2-я редакция. Отозван 06.05.1999.
3. ISO/IEC 8807. Information Processing Systems — Open Systems Interconnection. LOTOS — A Formal Description Technique Based on the Temporal Ordering of Observational Behaviour. Geneve, Switzerland: ISO, 1989. 142 p.
4. ITU-T X.680 (11/08) Information technology - Abstract Syntax Notation One (ASN.1): Specification of basic notation. ITU-T, 2008. 194 p.
5. OMG Unified Modeling LanguageTM (OMG UML), Infrastructure . OMG, 2011. 230 p.
6. Edmund M. Clarke, Orna Grumberg and Doron Peled. Model checking. MIT Press, 1999. 325 pp.
7. Karpov Yu. MODEL СHECKING. BHV-Peterburg, 2010, 560 pp.
8. M. Diaz. Petri Nets: Fundamental Models, Verification and Applications. Willey, 2013г. 656с.
9. Utting, M., Legeard, B.: Practical Model-Based Testing: A Tools Approach. Morgan Kaufmann, San Francisco (2007).
10. Blackburn, M., Busser, R., Nauman, A.: Why Model-Based Test Automation is Different and What You Should Know to Get Started. Software Productivity Consortium, NFP (2004).
11. Dalal, S.R., Jain, A., Karunanithi, N., Leaton, J.M., Lott, p.M., Patton, G.C., Horowitz, B.M.: Model-Based Testing in Practice. In: Proceedings of the ICSE 1999 (May 1999).
12. I. Bourdonov, A. Kossatchev, V. Kuliamin, A. Petrenko. UniTesK Test Suite Architecture. Proceedings of FME’2002, Kopenhagen, Denmark, LNCS 2391:77-88, Springer-Verlag, 2002.
13. V. V. Kuliamin, A. K. Petrenko, A. S. Kossatchev, and I. B. Burdonov. 2003. The UniTesK Approach to Designing Test Suites. Program. Comput. Softw. 29, 6 (November 2003), 310-322.
14. V. Kuliamin, A. Petrenko. Evolution of UniTESK Test Development Technology. ISP RAS Proceedings, vol. 26, issue 1, 2014. pp. 9-26. DOI: 10.15514/ISPRAS-2014-26(1)-1. (in Russian).
15. F. p. Hennie. Fault detecting experiments for sequential circuits. Proc. 5-th Ann. Symp. Switching Circuit Theory and Logical Design, 1964. p. 95-110.
16. M. Vasilevsky. Error diagnostics in state machines. Cybernetics and system analysis, vol. 9, № 4, 1973. pp. 98-108 (in Russian).
17. T. S. Chow. Testing software design modeled by finite-state machines. IEEE Trans. on Software Engineering, vol. 4, no. 3, 1978. p. 178-187.
18. J. A. Bergstra, J. W. Klop. Algebra of Communicating Processes with Absraction. Theoretical Computer Science, 37(1), 1985. pp. 77-121.
19. p. A. R. Hoare. Communicating Sequential Processes. Prentice-Hall, 1985; e-book http://www.usingcsp.com/cspbook.pdf, 2004. 260 p.
20. R. Milner. Communication and Concurrency. Prentice-Hall, 1989. 260 p.
21. N.A. Lynch, M.R. Tuttle, “An Introduction to Input/Output Automata” CWI-Quaterly 3, 1989, P. 219-246. [PDF] http://www.markrtuttle.com/papers/lt89-cwi.pdf.
22. F. Maraninchi. Operational and compositional semantics of synchronous automaton compositions. CONCUR'92 Proceedings, LNCS 630. Springer-Verlag, 1992.
23. ISO/IEC 7498. Information technology – Open Systems Interconnection – Basic Reference Model. Geneva, Switzerland: ISO, 1994.
24. Ed. Brinksma. A theory for the derivation of tests. Proc. IFIP WG6.1 8th Intl. Symp. on Protocol Specification, Testing, and Verification, North-Holland, S. Aggarwal and K. Sabnani Ed. pp. 63-74, 1988.
25. K. K. Sabnani and A. T. Dahbura. A protocol test generation procedure. Computer Networks and ISDN Systems, vol. 15, no. 4, pp. 285-297, 1988.
26. S. Fujiwara, G. v. Bochmann, F. Khendek, M. Amalou, and A. Ghedamsi. Test selection based on finite state models. IEEE Trans. on Software Eng., vol. 17, pp. 591-603,1991.
27. G. Luo, A. Petrenko, and G. v. Bochmann. Selecting test sequences for partially specified nondeterministic finite state machines, Proceedings of the IFIP Seventh International Workshop on Protocol Test Systems, Japan, 1994, pp. 95-110.
28. N. V. Evtushenko, A. V. Lebedev, A. F. Petrento, “Development of the checking set for a component of a sequential automaton network”. Automation and Remote Control, 1994, 55:8, 1203–1210
29. J. C. Fernandez, C. Jard, T. Jeron, C. Viho. Using on-the-Fly Verification Techniques for the Generation of Test Suites Proceedings of the 8th International Conference on Computer Aided Verification, LNCS 1102, Springer-Verlag, 1996, P. 348-359.
30. J. Tretmans. Test Generation with Inputs, Outputs, and Repetitive Quiescence. Software — Concepts and Tools, 17(3):103-120, 1996.
31. D. P. Sidhu and T.-K. Leung. Formal methods for protocol testing: a detailed study. IEEE Trans. Soft. Eng., vol. 15, no. 4, pp. 413-426, 1989.
32. D. Lee, M. Yannakakis. Principles and methods of testing finite state machines — a survey. Proc. IEEE, 84(8):1090-1123, 1996.
33. G. v. Bochmann, A. Petrenko. Protocol Testing: Review of Methods and Relevance for Software Testing. Proc. of ACM SIGSOFT ISSTA'1994, Software Engineering Notes, Special Issue, pp. 109—124.
34. A. Petrenko. Fault Model-Driven Test Derivation from Finite State Models: Annotated Bibliography. In F. Cassez, p. Jard, B. Rozov, M. Dermot, eds. Modeling and Verification of Parallel Processes: 4-th Summer School, Nantes, France, LNCS 2067, pp. 196-200, Springer-Verlag, 2000.
35. ISO/IEC 9646. Information technology – Open Systems Interconnection – Conformance testing methodology and framework – Part 1: General concepts. Geneva: ISO, 1994. 46 p.
36. ITU-T Recommendation Z.500. Framework on formal methods in conformance testing. Geneve, Switzerland: ITU, 1997. 49 p.
37. Glenford J. Myers, Corey Sandler, Tom Badgett. The Art of Software Testing, 3rd Edition. Wiley, 2011. p. 240.
38. J. Tretmans, An Overview of OSI Conformance Testing. Translated and adapted from: J. Tretmans and J. van de Lagemaat, Conformance Testen, in Handboek Telematica, Vol. II, pages 1--19. Samson, 1991.
39. ISO/IEC 9646. Information technology – Open Systems Interconnection – Conformance testing methodology and framework – Part 2: Abstract Test Suite specification. Geneva: ISO, 1994. 33 p.
40. Information technology – Open systems interconnection – Conformance testing methodology and framework – Part 3: The Tree and Tabular Combined Notation (TTCN). 1-е издание. Geneva, Switzerland: ISO, 1992.
41. Information technology – Open systems interconnection – Conformance testing methodology and framework – Part 3: The Tree and Tabular Combined Notation (TTCN). 2-е издание. Geneva, Switzerland: ISO, 1998.
42. ETSI ES 201 873-1 V3.1.1. Methods for Testing and Specification (MTS); The Testing and Test Control Notation version 3; Part 1: TTCN-3 Core Language. Sophia-Antipolis, France: ETSI, 2005. 210 p.
43. OMG formal/05-07-07. UML Testing Profile. Version 1.0. Needham, USA: Open Management Group, 2005. [PDF, PostScript] (http://www.omg.org/cgi-bin/doc?formal/05-07-07).
44. L. Ebrecht, M. Schacher, p. Bühler. Test Specification in XML – the most important Element for Test Automation. ARTiSAN Benutzerforum D.A.CH, 2005.
45. JUnit testing framework. http://www.junit.org.
46. IPv6 Test Suite TAHI. http://www.tahi.org/.
47. Digital cellular telecommunications system (Phase 2+) (GSM); Handover procedures (GSM 03.09 version 5.1.0) ETSI, Sophia-Antipolis, France, 1997. 81 p.
48. ETSI ETS 300 838. Integrated Services Digital Network (ISDN); Harmonized Programmable Communication Interface (HPCI) for ISDN. ETSI, Sophia-Antipolis, France, 1998. 546 p.
49. C. Jard, T. Jéron. TGV: Theory, principles and algorithms. International Journal on Software Tools for Technology Transfer (STTT), vol. 7(4). Berlin: Springer, 2005. p. 297 – 315
50. E. Farchi, A. Hartman, S.S. Pinter. Using a Model-based Test Generator to Test for Standard Conformance. IBM System Journal - special issue on Software Testing. Volume 41(1), 2002. p. 89 - 110.
51. G. Friedman, A. Hartman, K. Nagin, T. Shiran. Projected State Machine Coverage for Software Testing. Proceedings of ISSTA 2002 International Symposium on Software Testing and Analysis. New York, USA: ACM Press, 2002. p. 134 – 143.
52. Margus Veanes, Colin Campbell, Wolfgang Grieskamp, Wolfram Schulte, Nikolai Tillmann, and Lev Nachmanson, Model-Based Testing of Object-Oriented Reactive Systems with Spec Explorer, in Formal Methods and Testing, vol. 4949, pp. 39-76, Springer Verlag, 2008
53. A. Belinfante, J. Feenstra, R. de Vries, J. Tretmans, N. Goga, L. Feijs, S. Mauw, L. Heerink. Formal test automation: A simple experiment. G. Csopaki, S. Dibuz, K. Tarnay, editors. 12th Int. Workshop on Testing of Communicating Systems. Budapest, Hungary: Kluwer Academic Publishers, 1999. p. 179-196.
54. Axel Belinfante. JTorX: a Tool for On-Line Model-Driven Test Derivation and Execution. In: Tools and Algorithms for the Construction and Analysis of Systems, TACAS 2010. LNCS vol. 6015, pp. 266-270. Springer.
55. N. Goga. A probabilistic coverage for on-the-fly test generation algorithms. Automated Verification of Critical Systems (AVoCS '03), 2003.
56. N. V. Pakulin and A. V. Khoroshilov. 2007. Development of formal models and conformance testing for systems with asynchronous interfaces and telecommunications protocols. Program. Comput. Softw. 33, 6 (November 2007), 316-335.
57. V. V. Kuliamin, A. K. Petrenko, N. V. Pakoulin, A. S. Kossatchev, I. B. Bourdonov. Integration of Functional and Timed Testing of Real-time and Concurrent Systems. Proceedings of the 5-th International Conference on Perspectives of System Informatics, July 9-12, 2003, Novosibirsk, Russia; LNCS 2890, Springer, 2003, pp. 450-461.
58. V. V. Kuliamin, A. K. Petrenko, N. V. Pakoulin. Practical Approach to Specification and Conformance Testing of Distributed Network Applications. Proceedings of the 2-nd International Service Availability Symposium, April 25-26, 2005, Berlin, Germany; LNCS 3694, Springer, 2005, pp. 68-83.
59. V. V. Kuliamin, A. K. Petrenko, N. V. Pakoulin. Extended Design-by-Contract Approach to Specification and Conformance Testing of Distributed Software. Proceedings of the 9-th World Multiconference on Systemics, Cybernetics, and Informatics, Model Based Development and Testing Workshop, July 10-13, 2005, Orlando, Florida, USA, pp. 65-70
60. C. A. R. Hoare. An axiomatic basis for computer programming. Communications of the ACM, volume 12 №10, 1969. p. 576-580.
61. R. P. Draves, A. Mankin, B. D. Zill. Implementing IPv6 for Windows NT. Proceedings of the 2nd USENIX Windows NT Symposium, Seattle, WA, August 3–4, 1998
62. I. Agamirzian, S. Groshev, A. Khoroshilov, G. Kluchnikov, A. Kossatchev, V. Omeltchenko, N. Pakulin, A. Petrenko, V. Shnitman. Using formal methods for MSR IPv6 conformance testing. Proc. of Next Generation Internet, Yaroslavl, 2002. Pp. 29-33 (in Russian)
63. G. Kluchnikov, A. Kossatchev, A. Petrenko, N. Pakulin, V. Shnitman. Using formal methods in testing of an IPv6 implmentation. ISPRAS Proceedings, vol. 4, pp. 121-140, 2003 (in Russian).
64. G. Kluchnikov, A. Kossatchev, A. Petrenko, N. Pakulin, V. Shnitman. Using formal methods in testing of Mobile IPv6 . Proc. of Next Generation Internet-2, Yaroslavl, pp. 20-25, 2003 (in Russian).
65. D. Zatsepin, V. Shnitman. Aspects of applications of UniTESK to testing of mobility functions of IPv6 protocol. ISPRAS Proceedings, vol. 13(1), pp. 143-170, 2007 (in Russian)
66. CTesK 2.1: SeC Language Reference. ISPRAS, 2005. 167 p.
67. G. Kluchnikov, N. Pakulin, V. Shnitman. Automated testing of network services of Internet protocols. Proc. of Scientific services in Internet — 2005, pp. 168-170 (in Russian).
68. N. Pakulin. Applying formal methods to testing of implementations of complex modern protocols. Proc. of int. workshop «Go4IT — towards new Internet technologies» 2007, pp. 11-18 (in Russian).
69. A. Nikeshin, N. Pakulin, V. Shnitman. Aspects of testing of network-level security services of IPsec version 2. Proc. of Scientific services in Internet — 2008 (in Russian).
70. A. Nikeshin, N. Pakulin, V. Shnitman. Development of a test suite for verification of IPsec v2 implementations. ISPRAS Proceedings, vol. 18, 2010, pp. 151-182 (in Russian).
71. A. Nikeshin, N. Pakulin, V. Shnitman. Verification of security functions of IPsec v2. Programmirovanie, vol. 37 № 1. 2011. pp. 36-56 (in Russian)
72. Bourdonov I.B., Demakov A.V., Jarov A.A., Kossatchev A.S., Kuliamin V.V., Petrenko A.K. and Zelenov S.V. Java Specification Extension for Automated Test Development Proceedings of PSI’2001. Novosibirsk, Russia July 2-6 2001, LNCS 2244:301-307. Springer-Verlag, 2001.
73. A. Nikeshin, N. Pakulin, V. Shnitman. Test Suite development for verification of TLS security protocol. IPSRAS Proceedings, vol. 23, pp. 387-404. 2012. DOI: 10.15514/ISPRAS-2012-23-22. (in Russian).
74. A. Tugaenko. Conformance testing of extensible mail Internet protocols. Proceedings of Young Researchers Forum «Lomonosov 2010». (in Russian)
75. N. Pakulin, A. Tugaenko. Development of conformance test suites for email protocols. Proceedings of APPI-2209, pp 154-160 (in Russian)
76. A. Tugaenko. Method for conformance testing of extensible Internet protocols. Proceedings of Young Researchers Forum «Lomonosov 2011». (in Russian)
77. N. Pakulin, A. Tugaenko. Model Based Conformance Testing for Extensible Internet Protocols Proceedings of SYRCoSE 2011.
78. N. Pakulin, A. Tugaenko, V. Shnitman. Model-based testing of internet e-mail protocols. Proceedings of ISPRAS, vol 20, 2011. p. 125-141. (in Russian)
79. N. Pakulin, A. Tugaenko, V. Shnitman. Model-based testing of internet e-mail protocols. Programming and Computer Software, vol. 38, №5, 268-275. 2012
80. J. Jacky, "PyModel: Model-based testing in Python", Northwest Python Day 2010.
81. Errfix: model-based testing in Ruby. https://code.google.com/p/errfix/
82. I. Bourdonov, A. Kossatchev, A. Petrenko, and D. Galter. KVEST: Automated Generation of Test Suites from Formal Specifications. FM’99: Formal Methods. LNCS, volume 1708, Springer-Verlag, 1999, pp. 608–621.
83. A.Grinevich, V.Kuliamin, D.Markovcev, A.Petrenko, V.Rubanov, A.Khoroshilov. Using formal methods to ensure conformance for programming standards. // Proceedings of ISPRAS, №10, 2006.
84. V.Kuliamin, A.Petrenko, V.Rubanov, A.Khoroshilov. Formalization of interface standards and automated construction of conformance tests. Information technologies. 8:2-7, Moscow, New Technologies, 2007.
Review
For citations:
Pakulin N., Shnitman V., Nikeshin A. Automation of conformance testing for communication protocols. Proceedings of the Institute for System Programming of the RAS (Proceedings of ISP RAS). 2014;26(1):109-148. (In Russ.) https://doi.org/10.15514/ISPRAS-2014-26(1)-4
Email this article 