Protocol automata recovery method using binary code
https://doi.org/10.15514/ISPRAS-2022-34(5)-3
Abstract
Security analysis of network programs includes set of reverse engineering tasks of network protocols. Data formats restoring and implemented protocol automaton are the previous task issues. Unlike quite researched problem of formats restoring where there are lots of scientist’s papers, finding out the protocol's automaton program implementation looks like terra incognita and the cornerstone is a protocol state description currently undefined. There are two general ways to retrieve the implemented protocol automaton: an analysis of the network traces and looking into binary trace of the target application. This article offers a second one method. The first aim of the paper is the way to describe a mathematical model of a protocol automaton and a method for projecting it onto an executing application binary code. The second is concept of the protocol state definition and a principle to detect the states transitions based on some "global" binary trace objects, are described. Thirdly, there is suggested a protocol automaton precising manner by in-memory fuzzing based on a "floating" fork-server to manage states transitions. Finally, developed toolset's scheme and experiments on its using with a real VPN client, are shown.
About the Author
Ivan Vladimirovich SHARKOV
Ivannikov Institute for System Programming of the Russian Academy of Sciences
Russian Federation
Russian Federation
Researcher at the Compiler Technologies Department
References
1. SMACC – State Machine Asynchronous C++. Available at: https://smacc.dev/behavioral-vs-protocol-state-machines/#, accessed 11.10.2021.
2. Poll E. LangSec meets state machines. Presentation at the IT day at SWIFT, Belgium, 2017. Available at: https://www.cs.ru.nl/E.Poll/talks/SWIFT_2017.pdf, accessed 11.10.2021.
3. Székely G., Ládi G. et al. Protocol State Machine Reverse Engineering with a Teaching-Learning. Acta Cybernetica, vol. 25, issue 2, 2021, pp. 517-535.
4. Ládi G., Buttyán L., Holczer T. GrAMeFFSI: Graph analysis based message format and field semantics inference for binary protocols using recorded network traffic. Infocommunications Journal, vol. 12, issue 2, 2020, pp. 25-33.
5. Shahbaz M., Groz R. Inferring mealy machines. Lecture Notes in Computer Science, vol. 5850, 2009, pp. 207-222.
6. Sija B.D, Goo Y.-H. et al. A Survey of Automatic Protocol Reverse Engineering Approaches, Methods, and Tools on the Inputs and Outputs View. Security and Communication Networks, 2018, Article ID 8370341, 17 p.
7. Shevertalov M., Mancoridis S.. A reverse engineering tool for extracting protocols of networked applications. In Proc. of the 14th Working Conference on Reverse Engineering (WCRE ’07), 2007, pp. 229-238.
8. Xiao M.-M., Yu S.-Z., and Wang Y. Automatic network protocol automaton extraction. In Proc. of the 3rd International Conference on Network and System Security (NSS ’09), 2009, pp. 336-343.
9. Trifilo A., Burschka S., Biersack E. Traffic to protocol reverse engineering. In Proc. of the IEEE Symposium on Computational Intelligence for Security and Defense Applications, 2009, pp. 1-8.
10. Antunes J., Neves N. Building an automaton towards reverse protocol engineering. 2009. Available at: http://www.di.fc.ul.pt/∼nuno/PAPERS/INFORUM09.pdf, accessed 11.10.2021.
11. Antunes J., Neves N., Verissimo P. Reverse engineering of protocols from network traces. In Proc. of the 18th Working Conference on Reverse Engineering (WCRE ’11), 2011, pp. 169-178.
12. Wang Y., Zhang Z. et al. Inferring protocol state machine from network traces: a probabilistic approach. In Proc. of the 9th Applied Cryptography and Network Security International Conference (ACNS ’11), 2011, pp. 1-18.
13. Zhang Z., Wen Q.-Y., and Tang W. Mining protocol state machines by interactive grammar inference. In Proc. of the 2012 3rd International Conference on Digital Manufacturing and Automation (ICDMA ’12), 2012, pp. 524-527.
14. Laroche P., Burrows A., and Zincir-Heywood A.N. How far an evolutionary approach can go for protocol state analysis and discovery. In Proc. of the IEEE Congress on Evolutionary Computation (CEC ’13), 2013, pp. 3228-3235.
15. Meng F., Liu Y. et al. Inferring protocol state machine for binary communication protocol. In Proc. of the IEEE Workshop on Advanced Research and Technology in Industry Applications (WARTIA ’14), 2014, pp. 870-874.
16. Borisov N., Brumley D.J. et al. Generic application-level protocol analyzer and its language. MSR Technical Report MSR-TR-2005-133, 2005, 15 p.
17. Wang Y., Li X. et al. Biprominer: automatic mining of binary protocol features. In Proc. of the 12th International Conference on Parallel and Distributed Computing, Applications and Technologies (PDCAT ’11), 2011, pp. 179-184.
18. Bossert G., Guihéry F., and Hiet G. Towards automated protocol reverse engineering using semantic information. In Proc. of the 9th ACM Symposium on Information, Computer and Communications Security, 2014. Pp. 51-62.
19. Bossert G., Guihéry F. Reverse and simulate your enemy botnet C&C. Mapping a P2P Botnet with Netzob. In Proc. of the Black Hat Abu Dhabi Conference, 2012, 21 p.
20. Luo J.-Z. and Yu S.-Z. Position-based automatic reverse engineering of network protocols. Journal of Network and Computer Applications, vol. 36, issue 3, 2013, pp. 1070-1077.
21. Comparetti P.M., Wondracek G. et al. Prospex: protocol specification extraction. In Proc. of the 30th IEEE Symposium on Security and Privacy, 2009, pp. 110-125.
22. Xiao M.-M. and Luo Y.-P.. Automatic protocol reverse engineering using grammatical inference. IFS, vol. 32, no. 5, pp. 3585–3594, Apr. 2017, DOI: 10.3233/JIFS-169294.
23. Goo Y.-H., Shim K.-S. et al. Protocol Specification Extraction Based on Contiguous Sequential Pattern Algorithm. IEEE Access, vol. 7, 2019, pp. 36057-36074.
24. List of (automatic) protocol reverse engineering tools/methods/approaches for network protocols. Available at: https://github.com/techge/PRE-list, accessed 14.10.2021.
25. Costa M., Crowcroft J. et al. Vigilante: End-To-End Containment of Internet Worms. In Proc. of the 20th ACM Symposium on Operating Systems Principles (SOSP 2005), 2005, pp. 133-147.
26. Lang K.J. Faster Algorithms for Finding Minimal Consistent DFAs. Technical Report. NEC Research Institute, 1999, 19 p.
27. Needleman S. and Wunsch C. A General Method Applicable to the Search for Similarities in the Amino Acid Sequence of Two Proteins. Journal of Molecular Biology, vol. 48, issue 3, 1970, pp. 443-453.
28. Grammar-Mutator. Available at: https://github.com/AFLplusplus/Grammar-Mutator, accessed 22.09.2022.
29. AFLplusplus. Available at: https://github.com/AFLplusplus/AFLplusplus.git, accessed 20.09.2022.
Review
For citations:
SHARKOV I.V. Protocol automata recovery method using binary code. Proceedings of the Institute for System Programming of the RAS (Proceedings of ISP RAS). 2022;34(5):43-62. (In Russ.) https://doi.org/10.15514/ISPRAS-2022-34(5)-3
Email this article 