Auto-calibration and synchronization of camera and MEMS-sensors
https://doi.org/10.15514/ISPRAS-2018-30(4)-11
Abstract
This article describes our ongoing research on auto-calibration and synchronization of camera and MEMS-sensors. The research is applicable on any system that consists of camera and MEMS-sensors, such as gyroscope. The main task of our research is to find such parameters as the focal length of camera and the time offset between sensor timestamps and frame timestamps, which is caused by frame processing and encoding. This auto-calibration makes possible to scale computer vision algorithms (video stabilization, 3D reconstruction, video compression, augmented reality), which use frames and sensor’s data, to a wider range of devices equipped with a camera and MEMS-sensors. In addition, auto-calibration allows completely abstracting from the characteristics of a particular device and developing algorithms that work on different platforms (mobile platforms, embedded systems, action cameras) independently of concrete device’s characteristics as well. The article describes the general mathematical model needed to implement such a functionality using computer vision techniques and MEMS-sensors readings. The authors present a review and comparison of existing approaches to auto-calibration and propose own improvements for these methods, which increase the quality of previous works and applicable for a general model of video stabilization algorithm with MEMS-sensors.
About the Authors
A. R. Polyakov
Saint Petersburg State University
Russian Federation
Russian Federation
A. V. Kornilova
Saint Petersburg State University
Russian Federation
Russian Federation
I. A. Kirilenko
Saint Petersburg State University
Russian Federation
Russian Federation
References
1. S. Liu, M. Li, S. Zhu, and B. Zeng. Codingflow: Enable video coding for video stabilization. IEEE Transactions on Image Processing, vol. 26, Issue 7, Apr. 2017, pp. 3291-3302
2. M. Grundmann, V. Kwatra, and I. Essa. Auto-Directed Video Stabilization with Robust L1 Optimal Camera Paths. In Proc. of the 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2011), 2011, pp. 225-232
3. W.-C. Hu, C.-H. Chen, Y.-J. Su, and T.-H. Chang. Feature-based real-time video stabilization for vehicle video recorder system. Multimedia Tools and Applications, vol. 77, no. 5, Mar. 1, 2018, pp. 5107–5127
4. F. Liu, M. Gleicher, J. Wang, H. Jin, and A. Agarwala. Subspace video stabilization. ACM Transactions on Graphics, vol. 30, issue 1, 2011, pp. 4:1-4:10
5. D. Chatzopoulos, C. Bermejo, Z. Huang, and P. Hui. Mobile augmented reality survey: From where we are to where we go. IEEE Access, vol. 5, pp. 6917–6950, 2017
6. A. Bethencourt and L. Jaulin. 3d reconstruction using interval methods on the kinect device coupled with an IMU. International Journal of Advanced Robotic Systems, vol. 10, no. 2, 2013
7. J. Rambach, A. Pagani, S. Lampe, R. Reiser, M. Pancholi, and D. Stricker. Fusion of unsynchronized optical tracker and inertial sensor in EKF framework for in-car augmented reality delay reduction. In Proc. of the 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct), Oct. 2017, pp. 109–114
8. A. Karpenko, D. Jacobs, and J. Baek. Digital Video Stabilization and Rolling Shutter Correction using Gyroscopes. Stanford Tech Report CTSR 2011-03, 2011, 15 p.
9. S. Bell, A. Troccoli, and K. Pulli. A Non-Linear Filter for Gyroscope-Based Video Stabilization. Lecture Notes in Computer Science, vol 8692, 2014, pp. 294-308
10. Kornilova A.V., Kirilenko I.A., Zabelina N.I. Real-time digital video stabilization using MEMS-sensors. Trudy ISP RAN/Proc. ISP RAS, vol. 29, issue 4, 2017, pp. 73-86. DOI: 10.15514/ISPRAS-2017-29(4)-5
11. R. Szeliski. Computer Vision: Algorithms and Applications. Springer-Verlag London, 2010, 812 p.
12. J. Diebel. Representing Attitude: Euler Angles, Unit Quaternions, and Rotation Vectors. Matrix, 58, 2006, pp. 1-35
13. E. Rublee, V. Rabaud, K. Konolige, and G. Bradski. Orb: An efficient alternative to sift or surf. In Proc. of the 2011 International Conference on Computer Vision, Nov. 2011, pp. 2564–2571
14. D. G. Lowe. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, vol. 60, no. 2, pp. 91–110
15. E. Karami, S. Prasad, and M. S. Shehata. Image matching using sift, surf, BRIEF and ORB: performance comparison for distorted images. arXiv:1710.02726 [cs.CV], 2017
Review
For citations:
Polyakov A.R., Kornilova A.V., Kirilenko I.A. Auto-calibration and synchronization of camera and MEMS-sensors. Proceedings of the Institute for System Programming of the RAS (Proceedings of ISP RAS). 2018;30(4):169-182. https://doi.org/10.15514/ISPRAS-2018-30(4)-11
Email this article 