Numerical Simulation of Particulate Matter Transport in the Atmospheric Urban Boundary Layer Using Lagrangian Approach: Physical Problems and Parallel Implementation

This paper presents the results of the development of a numerical model of the Lagrangian particle transport and the application of parallel computation methods to increase the efficiency of the software implementation of the model. The model is a software package allowing calculations of transport and deposition of aerosol particles taking into account the properties of particles and input data describing atmospheric conditions and the underlying surface geometry. The dynamic core, physical parameterizations, numerical implementation, and algorithm of the model are described. Initially, the model has been used for computationally low-intensive problems. In this paper, given the need to use the model in computationally intensive problems, we conduct optimization of the sequential software implementation of the model, as well as creation of software implementations of the model with the use of parallel computing technologies OpenMP, MPI, CUDA. The results of testing of different implementations of the model show that optimization of the most computationally complex blocks in the sequential version of the model can reduce the execution time by 27%, at the same time the use of parallel computing technologies allows to achieve acceleration by several orders of magnitude. The use of OpenMP in dynamic block of the model resulted in acceleration of block up to 4 times, the use of MPI – up to 8 times, the use of CUDA – up to 16 times, all other conditions being equal. Recommendations on the choice of parallel computing technology depending on the properties of the computing system are proposed.

1. Thomson D. J., Wilson J. D. History of Lagrangian Stochastic Models for Turbulent Dispersion // Geo-physical Monograph Series / Lin J, Brunner D, Gerbig C, Stohl A, Luhar A, Webley P. Washing-ton, D. C. (eds.): American Geophysical Union, 2013. pp. 19–36. doi: 10.1029/2012GM001238.

2. Maronga B., Gryschka M., Heinze R., Hoffmann F., Kanani-Sühring F., Keck M., Ketelsen K., Letzel M. O., Sühring M., Raasch S. The Parallelized Large-Eddy Simulation Model (PALM) version 4.0 for atmospheric and oceanic flows: model formulation, re-cent developments, and future perspectives // Geosci. Model Dev. 2015, vol. 8(8), pp. 2515–2551. doi: 10.5194/gmd-8-2515-2015.

3. Huttner S. Further development and application of the 3D microclimate simulation ENVI-met // 2012.

4. Sofiev M., Vira J., Kouznetsov R., Prank M., Soares J., Genikhovich E. Construction of the SILAM Eu-lerian atmospheric dispersion model based on the advection algorithm of Michael Galperin // Geosci. Model Dev. 2015, vol. 8(11), pp. 3497–3522. doi: 10.5194/gmd-8-3497-2015.

5. Glazunov A., Rannik Ü., Stepanenko V., Lykosov V., Auvinen M., Vesala T., Mammarella I. Large-eddy simulation and stochastic modeling of Lagrangian particles for footprint determination in the stable boundary layer // Geosci. Model Dev. 2016, vol. 9(9), pp. 2925–2949. doi: 10.5194/gmd-9-2925-2016.

6. Auvinen M., Järvi L., Hellsten A., Rannik Ü., Vesala T. Numerical framework for the computation of urban flux footprints employing large-eddy simulation and Lagrangian stochastic modeling // Geosci. Model Dev. 2017. Т. 10. № 11. С. 4187–4205. https://doi.org/10.5194/gmd-10-4187-2017.

7. Simon H., Heusinger J., Sinsel T., Weber S., Bruse M. Implementation of a Lagrangian Stochastic Particle Trajectory Model (LaStTraM) to Simulate Concentration and Flux Footprints Using the Microclimate Model ENVI-Met // Atmosphere. 2021, vol. 12(8), pp. 977. doi: 10.3390/atmos12080977.

8. Stein A. F., Draxler R. R., Rolph G. D., Stunder B. J. B., Cohen M. D., Ngan F. NOAA’s HYSPLIT Atmospheric Transport and Dispersion Modeling System // Bulletin of the American Meteorological Society. 2015, vol. 96(12), pp. 2059–2077. doi: 10.1175/BAMS-D-14-00110.1.

9. Ansys Fluent. Theory Guide 12.0. [Электронный ресурс]. URL: https://www.afs.enea.it/project/neptunius/docs/fluent/html/th/main_pre.htm (access: 01.05.2023).

10. Morsi S. A., Alexander A. J. An investigation of particle trajectories in two-phase flow systems // J. Fluid Mech. 1972, vol. 55(02), 193 p. doi: 10.1017/S0022112072001806.

11. Fletcher N. H. The physics of rainclouds. Cambridge University Press // Quarterly Journal of the Royal Meteorological Society. 1962, № 378 (88). C. 559–559. doi: 10.1002/qj.49708837821.

12. Budd W. F. The Drifting of Nonuniform Snow Particles. Washington, D. C.: American Geophysical Union, 1966, pp. 59–70. doi: 10.1029/AR009p0059.

13. Wamser C., Lykossov V. N. On the friction velocity during blowing snow // Beitr Phys Atmosph. 1995, vol. 68(1), pp. 85–94. https://epic.awi.de/id/eprint/3270/.

14. Pope S. B. Turbulent Flows: Cambridge University Press, 2000, Issue 1. doi: 10.1017/CBO9780511840531.

15. Durbin P. A. Stochastic differential equations and turbulent dispersion // NASA, 1983. https://ntrs.nasa.gov/citations/19830014275.

16. Boughton B. A., Delaurentis J. M., Dunn W. E. A stochastic model of particle dispersion in the atmosphere // Boundary-Layer Meteorol. 1987. Т. 40. № 1–2. С. 147–163. https://doi.org/10.1007/BF00140073.

17. Reynolds A. M., Cohen J. E. Stochastic simulation of heavy-particle trajectories in turbulent flows // Physics of Fluids. 2002, vol. 14(1), pp. 342–351. doi: 10.1063/1.1426392.

18. Blocken B. LES over RANS in building simulation for outdoor and indoor applications: A foregone conclusion? // Build. Simul. 2018, vol. 11(5), pp. 821–870. doi: 10.1007/s12273-018-0459-3.

19. ENVI-met – Decoding urban nature. URL: https://www.envi-met.com/ (access: 01.05.2023).

20. Kadantsev E., Mortikov E., Zilitinkevich S. The resistance law for stably stratified atmospheric planetary boundary layers // Q J R Meteorol Soc. 2021, vol. 147(737). pp. 2233–2243. doi: 10.1002/qj.4019.

21. Tkachenko E. V., Debolskiy A. V., Mortikov E. V. Intercomparison of Subgrid Scale Models in Large-Eddy Simulation of Sunset Atmospheric Boundary Layer Turbulence: Computational Aspects // Lobachevskii J Math. 2021, vol. 42(7), pp. 1580–1595. doi: 10.1134/S1995080221070234.

22. Varentsov A. I., Stepanenko V. M., Mortikov E. V., Konstantinov P.I. Numerical simulation of particle transport in the urban boundary layer with implications for SARS-CoV-2 virion distribution // IOP Conf. Ser.: Earth Environ. Sci. 2020, vol. 611(1), pp. 012017. doi: 10.1088/1755-1315/611/1/012017.

23. Varentsov A. I., Stepanenko V. M., Konstantinov P. I. High-resolution simulation of particle transport in the urban atmospheric boundary layer // IOP Conf. Ser.: Earth Environ. Sci. 2019, vol. 386(1), 012045. doi: 10.1088/1755-1315/386/1/012045.

24. Glazunov A. V., Debolskiy A. V., Mortikov E. V. Turbulent Length Scale for Multilayer RANS Model of Urban Canopy and Its Evaluation Based on Large-Eddy Simulations // Supercomputing Frontiers and Innovations. 2021, vol. 8(4), pp. 100–116. doi: 10.14529/jsfi210409.

25. Glazunov A., Mortikov E., Debolskiy A. Studies of Stable Stratification Effect on Dynamic and Thermal Roughness Lengths of Urban-Type Canopy Using Large-Eddy Simulation // Journal of the Atmospheric Sciences. 2023, vol. 80(1), pp. 31–48. doi: 10.1175/JAS-D-22-0044.1.