Real-time digital video stabilization using MEMS-sensors

This article describes our ongoing research on real-time digital video stabilization using MEMS-sensors. The authors propose to use the described method for stabilizing the video that is transmitted to the mobile robot operator who controls the vehicle remotely, as well as increasing the precision of video-based navigation for subminiature autonomous models. The article describes the general mathematical models needed to implement the video stabilization module based on the MEMS sensors readings. These models includes the camera motion model, frame transformation model and rolling-shutter model. The existing approaches to stabilization using sensors data were analyzed and considered from the point of view of the application in a real-time mode. This article considers the main problems that came up during the experiments that were not resolved in the previous research papers. Such problems include: calibration of the camera and sensors, synchronization of the camera and sensors, increasing the accuracy of determining the camera position from sensors data. The authors offer possible solutions to these problems that would help improve quality of the work of existing algorithms, such as a system for parallel synchronized recording of video and sensor data based on the Android operating system. As the main result, the authors represent a framework for implementing video stabilization algorithms based on MEMS sensors readings.

video stabilization, MEMS sensors, real-time system, digital signal processing, computer vision, rolling shutter

1. Grundmann M., Kwatra V. and Essa I. Auto-directed video stabilization with robust L1 optimal camera paths, CVPR 2011, Providence, RI, 2011, pp. 225-232. DOI: 10.1109/CVPR.2011.5995525.

2. Y. Matsushita, E. Ofek, Weina Ge, Xiaoou Tang and Heung-Yeung Shum. Full-frame video stabilization with motion inpainting. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 28, no. 7, pp. 1150-1163, July 2006.

3. Feng Liu, Michael Gleicher, Jue Wang, Hailin Jin and Aseem Agarwala. Subspace Video Stabilization. ACM Transactions on Graphics (presented at SIGGRAPH 2011). Vol. 30, Issue 1, 2011: 4:1-4:10.

4. Y. S. Wang, F. Liu, P. S. Hsu and T. Y. Lee. Spatially and Temporally Optimized Video Stabilization. IEEE Transactions on Visualization and Computer Graphics, vol. 19, no. 8, pp. 1354-1361, Aug.2013. DOI: 10.1109/TVCG.2013.11.

5. S. Liu, L. Yuan, P. Tan, and J. Sun. Bundled camera paths for video stabilization. ACM Transactions on Graphics (TOG) - SIGGRAPH 2013 Conference Proceedings. Volume 32 Issue 4, July 2013. Article No. 78. DOI: 10.1145/2461912.2461995.

6. R. Szeliski. Computer Vision: Algorithms and Applications. 2010.

7. A. Karpenko. Digital Video Stabilization and Rolling Shutter Correction using Gyroscopes. Stanford Tech Report CTSR 2011-03. Stanford University.

8. Bell, S., Troccoli, A. J. & Pulli, K. (2014). A Non-Linear Filter for Gyroscope-Based Video Stabilization. In D. J. Fleet, T. Pajdla, B. Schiele & T. Tuytelaars (eds.), ECCV (4) (p./pp. 294-308), : Springer. ISBN: 978-3-319-10592-5.

9. S. Madgwick. An efficient orientation filter for inertial and inertial/magnetic sensor arrays. Report x-io and University of Bristol (UK), 2010.

10. R. Mahony, T. Hamel, J. M. Pflimlin. Complementary filter design on the special orthogonal group SO(3). Proceedings of the 44th IEEE Conference on Decision and Control, 2005, pp. 1477-1484. DOI: 10.1109/CDC.2005.1582367.

11. Rong Zhu, Dong Sun, Zhaoying Zhou, Dingqu Wang, A linear fusion algorithm for attitude determination using low cost MEMS-based sensors, Measurement, Volume 40, Issue 3, 2007, Pages 322-328, ISSN 0263-2241.

12. J. Diebel. Representing Attitude: Euler Angles, Unit Quaternions, and Rotation Vectors, 2006.

13. C. Jia and B. L. Evans. Online Camera-Gyroscope Autocalibration for Cell Phones. IEEE Transactions on Image Processing, vol. 23, no. 12, pp. 5070-5081, Dec. 2014. DOI: 10.1109/TIP.2014.2360120.

14. H. Ovrén and P. E. Forssén. Gyroscope-based video stabilisation with auto-calibration. 2015 IEEE International Conference on Robotics and Automation (ICRA), Seattle, WA, 2015, pp. 2090-2097. DOI: 10.1109/ICRA.2015.7139474.

15. Wang Qi, Fu Li and Liu Zhenzhong. Review on camera calibration. 2010 Chinese Control and Decision Conference, Xuzhou, 2010, pp. 3354-3358. DOI: 10.1109/CCDC.2010.5498574.