BSQ-rate: a new approach for video-codec performance comparison and drawbacks of current solutions
Anastasia Vsevolodovna Zvezdakova,
Dmitry Leonidovich Kulikov,
Sergey Vasilievitch Zvezdakov,
Dmitry Sergeevich Vatolin https://doi.org/10.15514/ISPRAS-2020-32(1)-5
Abstract
This paper is dedicated to the analysis of the existing approaches to video codecs comparisons. It includes the revealed drawbacks of popular comparison methods and proposes new techniques. The performed analysis of user-generated videos collection showed that two of the most popular open video collections from media.xiph.org which are widely used for video-codecs analysis and development do not cover real-life videos complexity distribution. A method for creating representative video sets covering all segments of user videos the spatial and temporal complexity is also proposed. One of the sections discusses video quality estimation algorithms used for video codec comparisons and shows the disadvantages of popular methods VMAF and NIQE. Also, the paper describes the drawbacks of the BD-rate – generally used method for video codecs final ranking during comparisons. A new ranking method called BSQ-rate which considers the identified issues is proposed. The results of this investigation were obtained during the series of research conducted as part of the annual video-codecs comparisons organized by video group of computer graphics and multimedia laboratory at Moscow State University.
Keywords
video quality,
no-reference metric,
quality measuring,
video-codec comparison,
comparison methodology,
bsq-rate
Supporting agencies: The studies were carried out with the financial support of RFBR in the framework of the scientific project 19-01-00785a. Special thanks go to George Osipov and Denis Kondranin, who helped analyze the problems found and improved the implementation of NIQE in the MSU VQMT software package, to Mikhail Erofeev, who participated in improving the video codec comparison methodology and made subjective comparisons, and the team of the Moscow State University of Computer Graphics and Multimedia Laboratory for valuable advice and support for our projects.
About the Authors
Anastasia Vsevolodovna Zvezdakova
Lomonosov Moscow State University
Russian Federation
Postgraduate student of the CMC faculty
Russian Federation
Postgraduate student of the CMC faculty
Dmitry Leonidovich Kulikov
Lomonosov Moscow State University,
Dubna State University
Russian Federation
Candidate of Physics and Mathematics, Associate Professor of the Institute for System Analysis and Management of Dubna State University, member of the video group of the Laboratory for Computer Graphics and Multimedia, Moscow State University
Russian Federation
Candidate of Physics and Mathematics, Associate Professor of the Institute for System Analysis and Management of Dubna State University, member of the video group of the Laboratory for Computer Graphics and Multimedia, Moscow State University
Sergey Vasilievitch Zvezdakov
Lomonosov Moscow State University
Russian Federation
Postgraduate student of the CMC faculty
Russian Federation
Postgraduate student of the CMC faculty
Dmitry Sergeevich Vatolin
Lomonosov Moscow State University
Russian Federation
Candidate of Physics and Mathematics, Senior Researcher, Laboratory of Computer Graphics and Multimedia
Russian Federation
Candidate of Physics and Mathematics, Senior Researcher, Laboratory of Computer Graphics and Multimedia
References
1. Cisco VNI Report 2017-2022, 2018 update. Available at: https://www.cisco.com/c/en/us/solutions/collateral/service-provider/visual-networking-index-vni/white-paper-c11-741490.html.
2. Xiph.org Test Media, 2019. Available at: https://media.xiph.org/.
3. Video Quality Experts Group Test Sequences, 2019. Available at: ftp://ftp.crc.ca/crc/vqeg/TestSequences/.
4. MPEG-2 Transport Stream Test Patterns and Tools, 2019. Available at: http://www.w6rz.net/.
5. Sveriges Television: The SVT High Definition Multi-Format Test Set, 2019. Available at: ftp://vqeg.its.bldrdoc.gov/HDTV/SVT_MultiFormat.
6. Columbia Consumer Video (CCV) Database, 2019. Available at: .http://www.ee.columbia.edu/ln/dvmm/CCV.
7. CDVL The Consumer Digital Video Library, 2019. Available at: https://www.cdvl.org/.
8. LIVE Public-Domain Subjective Video Quality Database, 2019. Available at: http://live.ece.utexas.edu/research/quality/live_video.html.
9. Video samples from KODI Wiki, 2019. Available at: https://kodi.wiki/view/Samples.
10. Ultra Video Group test sequences, 2019. Available at: http://ultravideo.cs.tut.fi/#testsequences
11. C. Chen, S. Inguva, A. Rankin, A. Kokaram. A subjective study for the design of multi-resolution ABR video streams with the vp9 codec. In Proc. of Electronic Imaging Symposium. Visual Information Processing and Communication VII, 2016, pp. 1-5.
12. D. Vatolin, D. Kulikov, M. Erofeev, A. Antsiferova, S. Zvezdakov, D. Kondranin, S. Grokholsky. 2019. MSU FullHD Video Codec Comparison 2019. Available at: http://compression.ru/video/codec_comparison/hevc_2019/#main_report.
13. MSU Video Codecs Comparisons, 2019. Available at: http://compression.ru/video/codec_comparison/index_en.html.
14. S. Chikkerur, V. Sundaram, M. Reisslein, L. J. Karam. Objective video quality assessment methods: A classification, review, and performance comparison. IEEE Transactions on Broadcasting, vol. 57, issue 2, 2011, pp. 165–182.
15. Z. Wang, A.C. Bovik, H.R. Sheikh, E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, vol. 13, issue 4, 2004, pp. 600–612.
16. H.R. Sheikh, A.C. Bovik. Image information and visual quality. IEEE Transactions on image processing, vol.15, issue 5, 2006, pp. 430-444.
17. VMAF: Perceptual video quality assessment based on multi-method fusion, 2016. Available at: https://medium.com/netflix-techblog/toward-a-practical-perceptual-video-quality-metric-653f208b9652
18. A. Mittal, R. Soundararajan, A. C. Bovik. Making a «completely blind» image quality analyzer. IEEE Signal Processing Letters, vol. 20, issue 3, 2012, p. 209-212.
19. D. Vatolin, D. Kulikov, M. Erofeev. MSU Video Codec Comparison 2017 Part III: Full HD Content, Subjective Evaluation, http://www.compression.ru/video/codec_comparison/hevc_2017/MSU_HEVC_comparison_2017_P3_subjective.pdf
20. K. Deb, A. Pratap, S. Agarwal, T. Meyarivan. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE transactions on evolutionary computation, vol. 6, issue 2, 2002, pp.182-197.
21. A. Zvezdakova, S. Zvezdakov, D. Kulikov, D. Vatolin. Hacking VMAF with Video Color and Contrast Distortion. In Proc. of the 29th International Conference on Computer Graphics and Vision (GraphiCon 2019). CEUR Workshop Proceedings, vol. 2485, 2019, pp. 53-57.
22. Ватолин Д.С., Гришин С.В. Двукратное увеличение частоты кадров видео на основе двунаправленной компенсации движения. Программирование, том 35, no. 6, 2009, стр. 67-80 / D.S. Vatolin, S. V. Grishin. Double up-conversion of video frame rate based on bidirectional motion compensation. Programming and Computer Software, vol. 35, no. 6, 2009, pp. 351-364.
23. D. Vatolin, D. Kulikov, M. Erofeev, A. Antsiferova, S. Zvezdakov, D. Kondranin. 2018. MSU Video Codec Comparison 2018, Subjective Report. Available at: http://compression.ru/video/codec_comparison/hevc_2018/#subjective_report.
24. A. Zvezdakova, D. Kulikov, D. Kondranin, D. Vatolin. Barriers Towards No-reference Metrics Application to Compressed Video Quality Analysis: on the Example of No-reference Metric NIQE. In Proc. of the 29th International Conference on Computer Graphics and Vision (GraphiCon 2019). CEUR Workshow Proceedings, 2019, Vol. 2485, p. 22-27.
25. G. Bjontegaard. Calculation of average PSNR differences between RD-curves. ITU-T VCEG, Document VCEG-M33, 2001.
Review
For citations:
Zvezdakova A.V., Kulikov D.L., Zvezdakov S.V., Vatolin D.S. BSQ-rate: a new approach for video-codec performance comparison and drawbacks of current solutions. Proceedings of the Institute for System Programming of the RAS (Proceedings of ISP RAS). 2020;32(1):89-108. (In Russ.) https://doi.org/10.15514/ISPRAS-2020-32(1)-5
Email this article 