Processing of raw astronomical data of large volume by MapReduce model
https://doi.org/10.15514/ISPRAS-2015-27(6)-20
Abstract
Exponential grow of volume, increased quality of data in current and incoming sky surveys open new horizons for astrophysics but require new approaches to data processing especially big data technologies and cloud computing. This work presents a MapReduce-based approach to solve a major and important computational task in astrophysics - raw astronomical image data processing.
About the Authors
S. . Gerasimov
Lomonosov Moscow State University Faculty CMC, 2nd Education Building
Russian Federation
Russian Federation
A. . Mesheryakov
Space Research Institute of the Russian Academy of Sciences
Russian Federation
Russian Federation
I. . Kolosov
Lomonosov Moscow State University Faculty CMC, 2nd Education Building
Russian Federation
Russian Federation
E. . Glotov
Lomonosov Moscow State University Faculty CMC, 2nd Education Building
Russian Federation
Russian Federation
I. . Popov
Lomonosov Moscow State University Faculty CMC, 2nd Education Building
Russian Federation
Russian Federation
References
1. The Sloan Digital Sky Survey (SDSS) http://www.sdss.org/
2. Burke B., Gregory J., Cooper M., Loomis A., Young D., Lind T., Doherty P., Daniels P., Landers D., Ciampi J., Johnson K., O’Brien P. CCD Imager Development for Astronomy. Lincoln Laboratory Journal, 2007, Volume 16, Number 2
3. Subaru-HSC http://www.naoj.org/Projects/HSC/
4. The Dark Energy Sky Survey (DES) http://www.darkenergysurvey.org/
5. Pan-STARRS http://pan-starrs.ifa.hawaii.edu/public/
6. The Large Synoptic Survey Telescope (LSST) http://www.lsst.org/
7. Zhang Y., Zhao Y. Astronomy in the Big Data Era. Data Science Journal, 2015
8. http://www.astromatic.net/
9. Wiley K., Connolly A., Gardner J., Krughof S., Balazinska M., Howe B., Kwon Y., Bu Y. Astronomy in the Cloud: Using MapReduce for Image Coaddition. Publications of the Astronomical Society of the Pacific, 2011, Vol. 123, No. 901, pp. 366-380
10. Apache Hadoop http://hadoop.apache.org
11. Montage: an astronomical image mosaic engine http://montage.ipac.caltech.edu/
12. Message Passing Interface Forum http://www.mpi-forum.org/
13. Pegasus: workflow management system http://pegasus.isi.edu/
14. Farivar R., Brunner R., Santucci R., Campbell R. Cloud Based Processing of Large Photometric Surveys. Astronomical Data Analysis Software and Systems XXII, 2013, p.91
15. Dean J., Ghemawat S. MapReduce: Simplified Data Processing on Large Clusters. OSDI'04: Sixth Symposium on Operating System Design and Implementation, December 2004
16. Koposov S., Belokurov V., Torrealba G., Wyn N. Evans Beasts of the Southern Wild: Discovery of nine Ultra Faint satellites in the vicinity of the Magellanic Clouds. The Astrophysical Journal, March 2015
17. Apache HBase http://hbase.apache.org
18. Apache Hive http://hive.apache.org
19. Apache Spark http://spark.apache.org
Review
For citations:
Gerasimov S., Mesheryakov A., Kolosov I., Glotov E., Popov I. Processing of raw astronomical data of large volume by MapReduce model. Proceedings of the Institute for System Programming of the RAS (Proceedings of ISP RAS). 2015;27(6):315-334. (In Russ.) https://doi.org/10.15514/ISPRAS-2015-27(6)-20
Email this article 