Overview of Hardening Mechanisms in Operating Systems and User Applications
Denis Valentinovich EFREMOV,
Alexander Konstantinovich PETRENKO,
Boris Aronovich POZIN,
Vitaly Adolfovich SEMENOV https://doi.org/10.15514/ISPRAS-2025-37(3)-23
Abstract
This paper presents a systematic review of hardening mechanisms for operating systems and user applications. Various types of protection mechanisms are discussed, including memory protection mechanisms, hardware stack protection, dynamic memory protection, address space randomization, control flow protection, and system integrity protection. The principles of these mechanisms, their effectiveness, and their impact on system performance are analyzed in detail. Special attention is given to the implementation of protective mechanisms in modern operating systems, particularly in the Linux kernel. This work is intended for information security specialists, operating system developers, and researchers working on information security issues.
Keywords
information security,
operating systems,
application security,
memory protection,
integrity control,
isolation,
security hardening,
hardening.
Supporting agencies: The work was supported by Kaspersky Lab as a part of the project “Analysis of world-class technology in architectural means of ensuring trust”.
About the Authors
Denis Valentinovich EFREMOV
Institute for System Programming of the Russian Academy of Sciences
Russian Federation
Russian Federation
Senior researcher. Research interests: formal verification, static and dynamic analysis.
Alexander Konstantinovich PETRENKO
Institute for System Programming of the Russian Academy of Sciences,
Lomonosov Moscow State University,
National Research University, Higher School of Economics
Russian Federation
Russian Federation
Dr. Sci. (Phys.-Math.), Prof., Head of Software Engineering Department of the Ivannikov Institute for System Programming, Russian Academy of Sciences, Professor of MSU and the Faculty of Computer Science, NUY HSE. Research their use in software development and verification.
Boris Aronovich POZIN
Institute for System Programming of the Russian Academy of Sciences, 3 National Research University, Higher School of Economics, “EC-leasing” Co.
Russian Federation
Russian Federation
Dr. Sci. (Tech.), Prof., Chief Researcher at the Ivannikov Institute for System Programming of the Russian Academy of Sciences, Professor of the Basic Department of CJSC EС-Leasing at the Higher School of Economics, Technical Director of CJSC EС-Leasing. Research interests: software engineering, life cycle ensuring systems for trusted software, automated software testing.
Vitaly Adolfovich SEMENOV
Institute for System Programming of the Russian Academy of Sciences, Moscow Institute of Physics and Technology
Russian Federation
Russian Federation
Dr. Sci. (Phys.-Math.), Prof., Head of the Department of System Integration and Multi-disciplinary Applied Systems of the Ivannikov Institute for System Programming of the RAS since 2015. Research interests: model-driven methodologies and CASE toolkits for creating digital platforms and advanced applied systems, visualization and computer graphics, building information modeling, project management and scheduling.
References
1. OpenBSD. Доступно по ссылке: https://www.openbsd.org/, 19.05.2025.
2. Linux Kernel Self-Protection Projection (KSPP). Доступно по ссылке: https://kspp.github.io/, 19.05.2025.
3. Rutkowska J., Wojtczuk R. Qubes OS architecture. Invisible Things Lab Tech Rep, 2010.
4. Grsecurity. Доступно по ссылке: https://grsecurity.net/, 19.05.2025.
5. PaX project. Доступно по ссылке: https://pax.grsecurity.net/docs/pax.txt, 19.05.2025.
6. GrapheneOS. Доступно по ссылке: https://grapheneos.org/features, 19.05.2025.
7. Common Weakness Enumeration: A community-developed list of SW & HW weaknesses that can become. Доступно по ссылке: https://cwe.mitre.org/index.html, 19.05.2025.
8. Попов А. Карта средств защиты ядра Linux. Системный Администратор, 2022, Выпуск №3 (232), Доступно по ссылке: https://samag.ru/archive/article/4535, 19.05.2025.
9. Xiong W., Lagerström R. Threat modeling – A systematic literature review. Comput. Secur., vol. 84, no. C, pp. 53–69, Jul. 2019, DOI: 10.1016/j.cose.2019.03.010.
10. Shevchenko N., Chick T. A., O’Riordan P., Scanlon T. P., Woody C. Threat Modeling: A Summary of Available Methods. 2018.
11. Freund J., Jones J. Measuring and Managing Information Risk: A FAIR Approach. Amsterdam: Butterworth-Heinemann, 2015, doi: https://doi.org/10.1016/B978-0-12-420231-3.00001-4.
12. Joint Task Force Transformation Initiative Risk management framework for information systems and organizations: a system life cycle approach for security and privacy. National Institute of Standards and Technology, Gaithersburg, MD, NIST SP 800-37r2, Dec. 2018. doi: 10.6028/NIST.SP.800-37r2.
13. International Organization for Standardization ISO/IEC 27005:2022 Information security, cybersecurity and privacy protection - Guidance on managing information security risks. 2022.
14. De Raadt T. Exploit Mitigation Techniques in OpenBSD. OpenCON, 2005, Доступно по ссылке: https://www.openbsd.org/papers/ven05-deraadt/index.html, 19.05.2025.
15. Pax Team Address Space Layout Randomization. 2001, Доступно по ссылке: https://pax.grsecurity.net/docs/aslr.txt, 19.05.2025.
16. Molnar I. x86: Enable KASLR by default. 2017, Доступно по ссылке: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=6807c84652b0b7e2e198e50a9ad47ef41b236e59, 19.05.2025.
17. De Raadt T. KARL - kernel address randomized link. 2017, Доступно по ссылке: https://marc.info/?l=openbsd-tech&m=149732026405941&w=2, 19.05.2025.
18. Molnar I. NX (No eXecute) support for x86. 2004, Доступно по ссылке: https://git.kernel.org/pub/scm/linux/kernel/git/history/history.git/commit/?id=36bc33bac78f6bc08282c622138f4e432b62e7be, 19.05.2025.
19. Pax Team PaX pageexec. 2000, Доступно по ссылке: https://pax.grsecurity.net/docs/pageexec.txt, 19.05.2025.
20. Pax Team PaX mprotect(). 2000, Доступно по ссылке: https://pax.grsecurity.net/docs/mprotect.txt, 19.05.2025.
21. De Raadt T. amd64 kernel W^X. 2015, Доступно по ссылке: https://marc.info/?l=openbsd-tech&m=142120787308107&w=2, 19.05.2025.
22. Zijlstra P. module: Harden STRICT_MODULE_RWX. 2020, Доступно по ссылке: https://lore.kernel.org/all/20200403171303.GK20760@hirez.programming.kicks-ass.net/, 19.05.2025.
23. Kemerlis V., Polychronakis M., Keromytis A. ret2dir: Rethinking kernel isolation. In 23rd USENIX Security Symposium (USENIX Security 14), pages 957–972, 2014.
24. Paris E. Allow Kconfig to set default mmap_min_addr protection. 2007, Доступно по ссылке: https://www.mail-archive.com/linux-security-module@vger.kernel.org/msg02399.html, 19.05.2025.
25. Intel Corporation INTEL64 AND IA-32 ARCHITECTURES SOFTWARE DEVELOPER’S MANUAL. Instruction Set Extensions Programming Reference. 2013.
26. George V., Piazza T., Jiang H.: Technology Insight: Intel Next Generation Microarchitecture. Codename Ivy Bridge, IDF 2011.
27. Yu F. Enable/Disable Supervisor Mode Execution Protection. 2011, Доступно по ссылке: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=de5397ad5b9ad22e2401c4dacdf1bb3b19c05679, 19.05.2025.
28. Corbet, J. Supervisor mode access prevention. 2012. Доступно по ссылке: http://lwn.net/Articles/517475/, 19.05.2025.
29. Spengler B. SSTIC 2016 Keynote. Rennes, France, 2016, Доступно по ссылке: https://grsecurity.net/SSTIC2016.pdf#page=3, 19.05.2025.
30. LWN Randomizing structure layout. 2017, Доступно по ссылке: https://lwn.net/Articles/722293/, 19.05.2025.
31. Serebryany, K., Bruening, D., Potapenko, A., & Vyukov, D. AddressSanitizer: A Fast Address Sanity Checker. USENIX ATC, 2012.
32. The Kernel Address Sanitizer (KASAN). Доступно по ссылке: www.kernel.org/doc/html/v4.10/dev-tools/kasan.html, 19.05.2025.
33. Elver M. KFENCE: A low-overhead sampling-based memory safety error detector. 2020, Доступно по ссылке: https://lore.kernel.org/all/20201103175841.3495947-1-elver@google.com/, 19.05.2025.
34. ARM, ARMv8.5-A Memory Tagging Extension.
35. Konovalov A. kasan: add hardware tag-based mode for arm64. 2020, Доступно по ссылке: https://lwn.net/Articles/838211/, 19.05.2025.
36. Zhao Q. Security Improvements in GCC. Linux Plumbers Conference, 2021, Доступно по ссылке: https://lpc.events/event/11/contributions/1001/, 19.05.2025.
37. Cook K. hardening: Introduce CONFIG_ZERO_CALL_USED_REGS. Доступно по ссылке: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=a82adfd5c7cb, 19.05.2025.
38. Jelinek J. Object size checking to prevent (some) buffer overflows. 2004, Доступно по ссылке: https://gcc.gnu.org/legacy-ml/gcc-patches/2004-09/msg02055.html, 19.05.2025.
39. Cook K. mm: Hardened usercopy. 2016, Доступно по ссылке: https://lwn.net/Articles/691012/, 19.05.2025.
40. Sidhpurwala H. Hardening ELF binaries using Relocation Read-Only (RELRO). Доступно по ссылке: https://www.redhat.com/en/blog/hardening-elf-binaries-using-relocation-read-only-relro, 19.05.2025.
41. Cowan C., Pu C., Maier D., Hintony H., Walpole J., Bakke P., Beattie S., Grier A., Wagle P., Zhang Q. StackGuard: automatic adaptive detection and prevention of buffer-overflow attacks. Proceedings of the 7th conference on USENIX Security Symposium, 1998.
42. Wagle P., Cowan C. StackGuard: Simple Stack Smash Protection for GCC. 2003.
43. Etoh H., Yoda K. Propolice: Protecting from stack-smashing attacks. Technical Report, IBM Research Division, Tokyo Research Laboratory, 2000.
44. Yu R., Nin F. D., Zhang Y., Huang S., Kaliyar P., Zakto S., Conti M., Portokalidis G., Xu J. Building Embedded Systems Like It’s 1996. Proceedings 2022 Network and Distributed System Security Symposium, DOI: 10.14722/ndss.2022.24031.
45. Bierbaumer, B., Kirsch, J., Kittel, T., Francillon, A., Zarras, A. Smashing the Stack Protector for Fun and Profit. SEC 2018, IFIP Advances in Information and Communication Technology, vol 529.2 2018 Springer, Cham. DOI: https://doi.org/10.1007/978-3-319-99828-2_21.
46. Depuydt H., Gülmez M., Nyman T., Mühlberg J. T. Do we still need canaries in the coal mine? Measuring shadow stack effectiveness in countering stack smashing. 2024, DOI: 10.48550/arXiv.2412.16343.
47. Delalleau G. Large memory management vulnerabilities. CanSecWest, 2005. Доступно по ссылке: https://cdn.atraining.ru/docs/memory_vulns_delalleau.pdf, 19.05.2025.
48. Wojtczuk R. Exploiting large memory management vulnerabilities in Xorg server running on Linux. 2010. Доступно по ссылке: https://invisiblethingslab.com/resources/misc-2010/xorg-large-memory-attacks.pdf, 19.05.2025.
49. Qualys Security Advisory: The Stack Clash. 2017. Доступно по ссылке: https://www.qualys.com/2017/06/19/stack-clash/stack-clash.txt, 19.05.2025.
50. Torvalds L. mm: keep a guard page below a grow-down stack segment. 2010. Доступно по ссылке: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=320b2b8de12698082609ebbc1a17165727f4c893, 19.05.2025.
51. Dickins H. mm: larger stack guard gap, between vmas. 2017. Доступно по ссылке: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1be7107fbe18eed3e319a6c3e83c78254b693acb, 19.05.2025.
52. Law J. Stack clash mitigation, 2017. Доступно по ссылке: https://gcc.gnu.org/legacy-ml/gcc-patches/2017-07/msg00556.html, 19.05.2025.
53. Guelton S., Ledru S., Stone J. Bringing Stack Clash Protection to Clang / X86 – the Open Source Way. 2021. Доступно по ссылке: https://blog.llvm.org/posts/2021-01-05-stack-clash-protection/, 19.05.2025.
54. Lutomirski A. Virtually mapped stacks with guard pages (x86, core). 2016. Доступно по ссылке: https://lkml.org/lkml/2016/6/15/1064, 19.05.2025.
55. Cook K. VLA removal for v4.20-rc1. 2018, Доступно по ссылке: https://lkml.org/lkml/2018/10/28/189, 19.05.2025.
56. Tomlin A. sched: Always check the integrity of the canary. Доступно по ссылке: https://lore.kernel.org/all/1410527779-8133-1-git-send-email-atomlin@redhat.com/, 19.05.2025.
57. Pax Team, PaX - kernel self-protection. H2HC, 2012. Доступно по ссылке: https://pax.grsecurity.net/docs/PaXTeam-H2HC12-PaX-kernel-self-protection.pdf, 19.05.2025.
58. Popov A. STACKLEAK: A Long Way to the Linux Kernel Mainline. Linux Security Summit 2018, Vancouber, Canada, 2018.
59. Cho H., Park J., Kang J., Bao T., Wang R., Shoshitaishvili Y., Doupé A., Ahn G.-J. Exploiting Uses of Uninitialized Stack Variables in Linux Kernels to Leak Kernel Pointers. WOOT 2020. Доступно по ссылке: https://www.usenix.org/conference/woot20/presentation/cho, 19.05.2025.
60. MISRA, MISRA C:2012 – Guidelines for the use of the C language in critical systems. MIRA Ltd, Nuneaton, Warwickshire CV10 0TU, UK (Mar 2013).
61. Pax Team, PaX - GCC plugins galore. H2HC, 2013. Доступно по ссылке: https://pax.grsecurity.net/docs/PaXTeam-H2HC13-PaX-gcc-plugins.pdf, 19.05.2025.
62. Cook K. gcc-plugins: Add structleak for more stack initialization. 2017, Доступно по ссылке: https://lore.kernel.org/kernel-hardening/20170113220256.GA57663@beast/T/#u, 19.05.2025.
63. Zhao Q. Security Improvements in GCC. Linux Plumbers Conference, 2021. Доступно по ссылке: https://lpc.events/event/11/contributions/1001/, 19.05.2025.
64. Zhao Q. add -ftrivial-auto-var-init and variable attribute uninitialized to gcc. GCC Patch Mailing List, February 2021, Доступно по ссылке: https://gcc.gnu.org/pipermail/gcc-patches/2021-February/565514.html, 19.05.2025.
65. Cook K. security: Implement Clang’s stack initialization. 2019. Доступно по ссылке: https://lore.kernel.org/all/20190411180117.27704-4-keescook@chromium.org/, 19.05.2025.
66. Pax Team, RANDKSTACK. Доступно по ссылке: https://pax.grsecurity.net/docs/randkstack.txt, 19.05.2025.
67. Reshetova E. Randomize kernel stack offset upon syscall. 2019, Доступно по ссылке: https://lwn.net/Articles/785484/, 19.05.2025.
68. Shacham, H. The Geometry of Innocent Flesh on the Bone: Return-into-libc without Function Calls. ACM CCS, 2007, DOI: 10.1145/1315245.1315313.
69. Garnier T. mm: SLAB freelist randomization. 2016, Доступно по ссылке: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c7ce4f60ac199fb3521c5fcd64da21cee801ec2b, 19.05.2025.
70. Cook K. mm: Add SLUB free list pointer obfuscation. 2017, Доступно по ссылке: https://lore.kernel.org/all/20170802180609.GA66807@beast/T/#m5fc98c10905ea25cf6123e723bca8162226d53c4, 19.05.2025.
71. Cook K. slab: Introduce dedicated bucket allocator. 2024, Доступно по ссылке: https://lwn.net/Articles/978976/, 19.05.2025.
72. Potapenko A. add init_on_alloc/init_on_free boot options. 2019, Доступно по ссылке: https://lore.kernel.org/all/20190628093131.199499-2-glider@google.com/T/#u, 19.05.2025.
73. Ruiqi G. Randomized slab caches for kmalloc(). 2023, Доступно по ссылке: https://lore.kernel.org/all/20230714064422.3305234-1-gongruiqi@huaweicloud.com/, 19.05.2025.
74. Abadi M., Budiu M., Erlingsson Ú., Ligatti J. Control-flow integrity. Proceedings of the 12th ACM conference on Computer and communications security. Alexandria VA USA: ACM, Nov. 2005, pp. 340–353. DOI: 10.1145/1102120.1102165.
75. Abadi M., Budiu M., Erlingsson Ú., Ligatti J. Control-flow integrity principles, implementations, and Applications. ACM Trans. Inf. Syst. Secur., vol. 13, no. 1, pp. 1–40, Oct. 2009, DOI: 10.1145/1609956.1609960.
76. Clercq R. D., Verbauwhede I. A survey of Hardware-based Control Flow Integrity (CFI). 2017.
77. Intel Corporation Control-flow Enforcement Technology Specification. 2021.
78. Shanbhogue V., Gupta D., Sahita R. Security Analysis of Processor Instruction Set Architecture for Enforcing Control-Flow Integrity. Proceedings of the 8th International Workshop on Hardware and Architectural Support for Security and Privacy. Phoenix AZ USA: ACM, Jun. 2019, pp. 1–11. DOI: 10.1145/3337167.3337175.
79. Tice C., Roeder T., Collingbourne P., Checkoway S., Erlingsson Ú., Lozano L., Pike G. Enforcing Forward-Edge Control-Flow Integrity in GCC & LLVM. USENIX Security 14, 2014, pp. 941–955. Доступно по ссылке: https://www.usenix.org/conference/usenixsecurity14/technical-sessions/presentation/tice, 19.05.2025.
80. Bento F. M. Control-Flow Integrity for the Linux kernel: A Security Evaluation. 2019.
81. Unified Extensible Firmware Interface Specification. Version 2.7, May 2017, Доступно по ссылке: https://uefi.org/sites/default/files/resources/UEFI_Spec_2_7.pdf, 19.05.2025.
82. Hagl J., Mann O., Pirker M. Securing the Linux Boot Process: From Start to Finish. Proceedings of the 7th International Conference on Information Systems Security and Privacy, 2021, pp. 604–610. DOI: 10.5220/0010313906040610.
83. Safford D. An Overview of The Linux Integrity Subsystem. Whitepaper, Доступно по ссылке: https://cyfuture.dl.sourceforge.net/project/linux-ima/linux-ima/Integrity_overview.pdf, 19.05.2025.
84. Zohar M. EVM. 2011, Доступно по ссылке: https://lore.kernel.org/all/1309377038-4550-1-git-send-email-zohar@linux.vnet.ibm.com/, 19.05.2025.
85. Garrett M. security: Add a static lockdown policy LSM. Доступно по ссылке: https://lore.kernel.org/linux-security-module/20190820001805.241928-4-matthewgarrett@google.com/, 19.05.2025.
86. Edge J. A seccomp overview, 2015. Доступно по ссылке: https://lwn.net/Articles/656307/, 19.05.2025.
87. Arcangeli A. seccomp for 2.6.11-rc1-bk8. 2005, Доступно по ссылке: https://lwn.net/Articles/120192/, 19.05.2025.
88. Salaün M. Landlock: From a security mechanism idea to a widely available implementation. 2024, Доступно по ссылке: https://landlock.io/talks/2024-06-06_landlock-article.pdf, 19.05.2025.
89. Salaün M. Landlock LSM. 2021, Доступно по ссылке: https://lore.kernel.org/linux-security-module/20210422154123.13086-1-mic@digikod.net/, 19.05.2025.
Review
For citations:
EFREMOV D.V., PETRENKO A.K., POZIN B.A., SEMENOV V.A. Overview of Hardening Mechanisms in Operating Systems and User Applications. Proceedings of the Institute for System Programming of the RAS (Proceedings of ISP RAS). 2025;37(3):325-354. (In Russ.) https://doi.org/10.15514/ISPRAS-2025-37(3)-23
Email this article 