Efficient methods and algorithms to synthesize 360-degree video based on cubemap projection of virtual environment

The paper deals with the task of creation and playback of panoramic video with 360-degree overview, which allows the researcher to be immersed in virtual medium outside parent virtual environment system (VES). To solve the task, the extension of the cubemap method is proposed, in which cubemap resolution is determined taking into account viewer camera field of view and screen resolution (Adequate Cubemap Projection, ACMP). The paper studies the influence of the camera orientation inside the cube on the "cubemap pixel / screen pixel" ratio determining panorama visualization quality. Based on this, a method to calculate cubemap resolution for high-quality panorama visualization for all possible camera orientations is proposed. The paper considers an efficient method and algorithm to create ACMP-video on the GPU using render-to-texture technology, which allow to synthesize panoramas with constant orientation or bound to the observer's view direction. In the research efficient methods and algorithms to play ACMP-video are also proposed, which are based on the visualization of visible cube faces and adaptive frame buffering. The obtained methods and algorithms are implemented in ACMP-video synthesis program complex (С++, OpenGL, FFmpeg) including frame capture module (embeddable into VES) and the player. The developed solution was tested in system «Virtual Earth» designed for training to observe Earth objects from the International Space Station (ISS). Using the capture module, an ACMP-video of the flight along the ISS orbit track was created. When playing this video, the trainee flies in orbit above virtual 3D surface of the Earth and can explore it by means of camera rotation. Testing of the complex confirmed the adequacy of the developed methods and algorithms to the task. The obtained scientific and practical results expand the capabilities and scope of application of VES, scientific visualization systems, video simulators and virtual laboratories; provide effective exchange of experience between researchers, etc.

360-degree video, omnidirectional video, cubemap projection, visualization, virtual environment

1. Михайлюк М.В., Мальцев А.В., Тимохин П.Ю., Страшнов Е.В., Крючков Б.И., Усов В.М. Системы виртуального окружения для прототипирования на моделирующих стендах использования космических роботов в пилотируемых полетах. Пилотируемые полеты в космос, том 35, № 2, 2020 г., стр. 61-75. / Mikhaylyuk M.V., Maltsev A.V., Timokhin P.Yu., Strashnov E.V., Kryuchkov B.I., Usov V.M. Virtual Environment Systems for Simulating Robots in Manned Space Fligts. Pilotiruemye polety v kosmos. Manned Spaceflight, vol. 35, № 2, 2020, pp. 61-75 (in Russian).

2. Барладян Б.Х., Шапиро Л.З., Маллачиев К.A., Хорошилов А.В., Солоделов Ю.А., Волобой А.Г., Галактионов В.А., Ковернинский И.В. Система визуализации для авиационной ОС реального времени JetOS. Труды ИСП РАН, том 32, вып. 1, 2020 г., стр. 57-70. DOI: 10.15514/ISPRAS-2020-32(1)-3 / Barladian B.Kh., Shapiro L.Z., Mallachiev K.A., Khoroshilov A.V., Solodelov Y.A., Voloboy A.G., Galaktionov V.A., Koverninskiy I.V. Rendering System for the Aircraft Real-Time OS JetOS. Trudy ISP RAN/Proc. ISP RAS, vol. 32, issue 1, 2020, pp. 57-70 (in Russian).

3. Михайлюк М.В., Торгашев М.А. Система визуализации “GLView” для имитационно-тренажерных комплексов и систем виртуального окружения. Труды 25-й Международной научной конференции GraphiCon, 2015, стр. 96-101 / Mikhaylyuk M.V., Torgashev M.A. The System of Visualization “GLView” for Simulators and Virtual Environment Systems. In Proc. of the 25th International Conference on Computer Graphics and Vision (GraphiCon 2015), 2015, pp. 96-101 (in Russian).

4. Страшнов Е.В., Мироненко И.Н., Финагин Л.А. Моделирование режимов полета квадрокоптера в системах виртуального окружения. Информационные технологии и вычислительные системы, № 1, 2020 г., стр. 85-94 / Strashnov E.V., Mironenko I.N., Finagin L.A. Simulation of quadcopter flight modes in virtual environment systems. Informacionnye tekhnologii i vichslitel’nye sistemy (Journal of Information Technologies and Computing Systems), № 1, 2020. pp. 85-94 (in Russian).

5. Тимохин П.Ю., Михайлюк М.В., Вожегов Е.М., Пантелей К.Д. Технология и методы отложенного синтеза 4K-стереороликов для сложных динамических виртуальных сцен. Труды ИСП РАН, том 31, вып. 4, 2019 г., стр. 61-72. DOI: 10.15514/ISPRAS-2019-31(4)-4. / Timokhin P.Yu., Mikhaylyuk M.V., Vozhegov E.M., Panteley K.D. Technology and methods for deferred synthesis of 4K stereo clips for complex dynamic virtual scenes. Trudy ISP RAN/Proc. ISP RAS, vol. 31, issue 4, 2019, pp. 61-72 (in Russian).

6. El-Ganainy T., Hefeeda M. Streaming Virtual Reality Content. arXiv:1612.08350, 2016..

7. Ray B., Jung J., Larabi M.-C. A Low-Complexity Video Encoder for Equirectangular Projected 360 Video Content. In Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2018, pp. 1723-1727

8. Li J., Wen Z., Li S., Zhao Y., Guo B., Wen J. Novel tile segmentation scheme for omnidirectional video. In Proc. of the IEEE International Conference on Image Processing (ICIP), 2016, pp. 370-374.

9. K.-T. Ng, S.-C. Chan, H.-Y. Shum. Data Compression and Transmission Aspects of Panoramic Videos. IEEE Transactions on Circuits and Systems for Video Technology, vol. 15, № 1, 2005, pp. 82-95

10. Brown C. Bringing pixels front and center in VR video. Google AR and VR, March 14, 2017. Available at: https://www.blog.google/products/google-ar-vr/bringing-pixels-front-and-center-vr-video/, accessed 18.03.2020.

11. Kuzyakov E., Liu S., Pio D. Optimizing 360 Video for Oculus. Facebook F8 developers conference, 2016. Available at: https://developers.facebook.com/videos/f8-2016/optimizing-360-video-for-oculus/, accessed 18.03.2020.

12. Chen Z., Wang X., Zhou Y., Zou L., Jiang J. Content-Aware Cubemap Projection for Panoramic Image via Deep Q-Learning. Lecture Notes in Computer Science, vol. 11962, 2020, pp. 304-315.

13. Fu C.-W., Wan L., Wong T.-T., Leung C.-S. The Rhombic Dodecahedron Map: An Efficient Scheme for Encoding Panoramic Video. IEEE Transactions on Multimedia, vol. 11, № 4, 2009, pp. 634-644.

14. Kuzyakov E., Pio D. Next-generation video encoding techniques for 360 video and VR. Available at: https://code.facebook.com/posts/1126354007399553/next-generation-video-encodin, accessed 18.03.2020.

15. Segal M., Akeley K. The OpenGL Graphics System: A Specification. Version 4.6, Core Profile. The Khronos Group Inc., 2006-2018. Available at: https://www.khronos.org/registry/OpenGL/specs/gl/ glspec46.core.pdf, accessed 18.03.2020.

16. FFmpeg. A complete, cross-platform solution to record, convert and stream audio and video. Available at: https://ffmpeg.org/, accessed 18.03.2020.

17. Timokhin P.Y., Mikhaylyuk M.V. Effective technology to visualize virtual environment using 360-degree video based on cubemap projection. In Proc. of International Conference on Computing for Physics and Technology (CPT2020), 2020.

18. Тимохин П.Ю. Моделирование видимого движения Земли вдоль участков суточной трассы МКС в космических видеотренажерах. Труды НИИСИ РАН, том. 9, № 6, 2019 г., стр. 111-117. DOI: 10.25682/NIISI.2019.6.0014 / Timokhin P.Yu. Simulation of visible Earth motion along daily tracks of ISS orbit in space simulators. Trudy NIISI RAN/Proc. of SRISA RAS, vol. 9, № 6, 2019, pp. 111-117 (in Russian).