Ways to organize parallel access to structured data
Alexey Olegovich IGNATYEV,
Sergey Yurievich MOKSHIN,
Dmitry Vladimirovich IVANKOV,
Evgeny Alexandrovich BEKETOV https://doi.org/10.15514/ISPRAS-2023-35(2)-8
Abstract
This paper explores ways to achieve the highest possible exchange performance with files containing structured data. The research was carried out on file systems with supercomputer systems parallel access designed to solve problems of physical and mathematical modeling of various processes and objects. For example, parallel access to raw data is considered using the Lustre file system. The article suggests a way to organize parallel access to structured data based on a specially developed PSIO storage format and the psio access library. A comparative analysis of the I/O performance of the developed data storage format and the HDF5 parallel version format is performed.
Keywords
computing system,
structured data,
high-performance computing simulation,
mathematical simulation subsystem,
data format,
HDF,
HPC,
SIO
Supporting agencies: The authors express special gratitude to R.M. Shagaliev and V.A. Sviridov for the NCFM Polygon computing resources provided for the tests (Sarov city).
About the Authors
Alexey Olegovich IGNATYEV
E. I. Zababakhin All-Russian Scientific Research Institute of Technical Physics
Russian Federation
Russian Federation
Head of Laboratory
Sergey Yurievich MOKSHIN
E. I. Zababakhin All-Russian Scientific Research Institute of Technical Physics
Russian Federation
Russian Federation
Head of Department
Dmitry Vladimirovich IVANKOV
E. I. Zababakhin All-Russian Scientific Research Institute of Technical Physics
Russian Federation
Russian Federation
Head of Laboratory
Evgeny Alexandrovich BEKETOV
E. I. Zababakhin All-Russian Scientific Research Institute of Technical Physics
Russian Federation
Russian Federation
Head of the work group
References
1. . HDF Group. Available at: https://www.hdfgroup.org/solutions/hdf5/, accessed 10.05.2023.
2. . Lustre. Available at: https://www.lustre.org, accessed 10.05.2023.
3. . The IO500 benchmark. Available at: http://www.io500.org, accessed 10.05.2023.
4. . Users Guide for ROMIO: A High-Performance, Portable MPI-IO Implementaion. Argonne National Laboratory. 9700 South Gass Avenue, Argonne, IL, 60439/ Revised May 2004. Available at: https://web.cels.anl.gov/~thakur/papers/users-guide.pdf ,accesed 10.05.2022
5. . NCPHM Polygon. Available at: https://ncphm.ru/, accessed 10.05.2023.
6. . Chunking in HDF5. Available at: https://support.hdfgroup.org/HDF5/doc/Advanced/Chunking/, accessed 10.05.2023.
7. . Tuning HDF5 for Lustre File System. Mark Howison, Quincey Koziol, David Knaak, John Mainzer, John Shalf. Available at: https://www.hdfgroup.org/archive/support/pubs/papers/howison_hdf5_lustre_iasds2010.pdf, accessed 10.05.2023.
Review
For citations:
IGNATYEV A.O., MOKSHIN S.Yu., IVANKOV D.V., BEKETOV E.A. Ways to organize parallel access to structured data. Proceedings of the Institute for System Programming of the RAS (Proceedings of ISP RAS). 2023;35(2):111-126. (In Russ.) https://doi.org/10.15514/ISPRAS-2023-35(2)-8
Email this article 