Dynamic analysis of virtualization- and dispatching-obfuscated applications
https://doi.org/10.15514/ISPRAS-2012-23-3
Abstract
Obfuscation algorithms are now widely used to prevent software reverse engineering. Binary code virtualization is one of the most powerful obfuscations technics. Another obfuscation method known as “dispatching” can be used to transform application control flow similarly to virtual machine insertion. Our research was aimed at reconstruction of control flow graph in case of both code virtualization and dispatching. To achieve this goal, we implemented de-obfuscation tool which keeps track of virtual program counter used by virtual machine emulator and reconstructs the application control flow. This paper describes experimental results of test application de-obfuscation via dynamic analysis. Both obfuscating and de-obfuscating tools were independently developed by two different teams of ISP RAS – the LLVM-based obfuscating compiler and the software environment for dynamic analysis of binary code. The paper briefly introduces both software tools and then describes results of experimental research on recovering of control flow graph of obfuscated application. Application was initially protected by specialized obfuscating LLVM-based compiler. Next, TrEx environment was used to analyze program execution trace, to find the dispatcher-protected part of application and to recover its control flow. Additionally, some software code complexity metrics for test applications were calculated to estimate obfuscation resilience provided by different versions of obfuscating compiler.
About the Authors
M. G. Bakulin
ISP RAS, Moscow
Russian Federation
Russian Federation
S. S. Gaissaryan
ISP RAS, Moscow
Russian Federation
Russian Federation
Sh. F. Kurmangaleev
ISP RAS, Moscow
Russian Federation
Russian Federation
I. N. Ledovskikh
ISP RAS, Moscow
Russian Federation
Russian Federation
V. A. Padaryan
ISP RAS, Moscow
Russian Federation
Russian Federation
S. M. Shchevyeva
ISP RAS, Moscow
Russian Federation
Russian Federation
References
1. Tikhonov A.YU., Avetisyan A.I. Razvitie taint-analiza dlya resheniya zadachi poiska programmnykh zakladok [Development of taint-analysis methods to solve the problem of searching of undeclared fatures]. Trudy ISP RAN [The Proceedings of ISP RAS], 2011, vol. 20, pp. 9–24 (in Russian).
2. Padaryan V. А., Get'man А. I., Solov'yov M. А. Programmnaya sreda dlya dinamicheskogo analiza binarnogo koda [Software environment for dynamic analysis of binary code]. Trudy ISP RAN [The Proceedings of ISP RAS], 2009, vol. 16, pp. 51–72 (in Russian).
3. Batuzov K., Dovgalyuk P., Koshelev V., Padaryan V. Dva sposoba organizatsii mekhanizma polnosistemnogo determinirovannogo vosproizvedeniya v simulyatore QEMU [Two approaches to full-system deterministic replay in QEMU]. Trudy ISP RAN [The Proceedings of ISP RAS], 2012, vol. 22, pp. 77–94 (in Russian).
4. Korel B., Laski J. Dynamic program slicing. Information Processing Letters, 1988, vol. 29, Issue 3, pp. 155–163. doi: 10.1016/0020-0190(88)90054-3
5. Get'man A.I., Markin YU.V., Padaryan V.A., SHHetinin E.I. Vosstanovlenie formata dannykh [Format recovery] Trudy ISP RAN [The Proceedings of ISP RAS], 2010, vol. 19, pp. 195–214 (in Russian).
6. Аvetisyan A.I., Get'man A.I. Vosstanovlenie struktury binarnykh dannykh po trassam program [Recovery of binary data structures from program traces] Trudy ISP RAN [The Proceedings of ISP RAS], 2012, vol. 22, pp. 95–118 (in Russian).
7. Sharif M., Lanzi A., Giffin J., Lee W. Automatic reverse engineering of malware emulators Proceedings of the 2009 30th IEEE Symposium on Security and Privacy — SP ’09. — Washington, DC, USA : IEEE Computer Society, 2009, pp. 94–109. doi: 10.1109/SP.2009.27
8. Ellson J., Gansner E.R., Koutsofios E., et al. Graphviz and dynagraph – static and dynamic graph drawing tools. Graph Drawing Software (ed. by M. Junger, P. Mutzel), Berlin/Heidelberg: Springer-Verlag, 2004. — Mathematics and Visualization. pp. 127–148. doi: 10.1007/978-3-642-18638-7_6
9. Ledovskikh I.N., Bakulin M.G.. Podkhod k vosstanovleniyu potoka upravleniya zaputannoj programmy [An approach to reconstruction of control flow of an obfuscated program] Trudy ISP RAN [The Proceedings of ISP RAS], 2012, vol. 22, pp. 153–168 (in Russian).
10. Kurmangaleev SH.F., Korchagin V.P., Matevosyan R.А. Opisanie podkhoda k razrabotke obfustsiruyushhego kompilyatora [Description of the Approach to Development of the Obfuscating Compiler] Trudy ISP RAN [The Proceedings of ISP RAS], 2012, vol. 23, pp. 67–76 (in Russian).
11. Kurmangaleev SH.F., Korchagin V.P., Savchenko V.V., Sargsyan S.S. Postroenie obfustsiruyushhego kompilyatora na osnove infrastruktury LLVM [Building Obfuscation Compiler Based on LLVM Infrastructure] Trudy ISP RAN [The Proceedings of ISP RAS], 2012, vol. 23, pp. 77–92 (in Russian).
12. Tikhonov А. YU., Аvetisyan А. I., Padaryan V. А. Metodika izvlecheniya algoritma iz binarnogo koda na osnove dinamicheskogo analiza [Methodology of Exploring of an Algorithm from Binary Code by Dynamic Analysis] Problemy informatsionnoj bezopasnosti. Komp'yuternye sistemy [Problems of Computer Security. Computer Systems], 2008, vol. 3, pp. 66–71 (in Russian).
13. Milyutin А., Metriki koda programmnogo obespecheniya [Software code metrics] http://www.viva64.com/ru/a/0045/ [http://www.viva64.com/en/a/0045/], 20.07.2009.
14. Hassan R. B. Automatic Measurement of Source Code Complexity. Master’s Thesis, Lulea University of Technology, Lulea, Sweden, 2011.
Review
For citations:
Bakulin M.G., Gaissaryan S.S., Kurmangaleev Sh.F., Ledovskikh I.N., Padaryan V.A., Shchevyeva S.M. Dynamic analysis of virtualization- and dispatching-obfuscated applications. Proceedings of the Institute for System Programming of the RAS (Proceedings of ISP RAS). 2012;23. (In Russ.) https://doi.org/10.15514/ISPRAS-2012-23-3
Email this article 