Integrated Application of Computer Vision and Speech Recognition to Improve the Quality of Processing Video Recordings of Production Works

Artificial intelligence technologies are increasingly applied to the scientific organization of labor, in particular to the technological processes of the railway industry. The primary benefit in this context comes from automating the routine human work of reviewing and analyzing video recordings of production operations. To realize this benefit, an initial method based on computer-vision techniques and structured-information processing was developed, and has since underpinned a functioning software. Industrial exploitation of the feature revealed the need for enhanced capabilities and improved performance. In response, the authors have devised an advanced approach that jointly employs a complex of computer-vision and speech-recognition technologies. The proposed solution includes employee detection and tracking, recognition of specialized tools and worker actions, and processing of the camera operator’s verbal commentary. By integrating data from multiple information channels, it increases the accuracy of temporal-interval determination for technological operations, compensating for the limitations of single-channel analysis. Laboratory experiments demonstrate that the use of modern neural-network architectures and the newly proposed post-processing algorithms delivers a 40% gain in the quality of video-analysis of production workflows.

1. Никитин В.Н. Научная организация труда в парадигме устойчивого развития транспортного комплекса. Транспортное дело России, №5, 2024 г., стр. 60-65. / Nikitin V.N. Scientific organization of labor in the paradigm of sustainable development of the transport complex. Transport business of Russia, no. 5, 2024, pp. 60-65.

2. Коптева Ж.Ю., Томакова И.А, Садиков Р.А. Современные информационные технологии изучения затрат рабочего времени как важный инструмент для эффективного управления бизнес-процессами. Известия Юго-Западного государственного университета. Серия: Экономика. Социология. Менеджмент, том 14, № 4, 2024 г., стр. 202–217. DOI: 10.21869/2223-1552-2024-14-4-202-217. / Kopteva Zh.Yu., Tomakova I.A., Sadikov R.A. Modern information technologies for studying the cost of working time as an important tool for effective management of business processes. Proceedings of the Southwest State University. Series: Economics. Sociology. Management, vol. 14, no. 4, 2024, pp. 202-217. (in Russian) DOI: 10.21869/2223-1552-2024-14-4-202-217.

3. Никитин В.Н. Новая научная организация труда как базис системы нормирования труда в ОАО «РЖД». Повышение производительности труда на транспорте – источник развития и конкурентоспособности национальной экономики: Труды V Всероссийской научно-практической конференции (г. Москва, 26 мая 2021 г.). Москва: Гуманитарный институт федерального государственного бюджетного образовательного учреждения высшего образования "Российский университет транспорта (МИИТ)", 2021 г., стр. 28–37.

4. Клемышев И.М., Колчин С.М., Лебедев С.С., Старков С.О. Автономная многоракурсная нейросетевая система компьютерного зрения для автоматизации контроля ручных операций. Успехи кибернетики, том 5, №3, 2024 г., стр. 24–33. DOI: 10.51790/2712-9942-2024-5-3-03. / Klemyshev I.M., Kolchin S.M., Lebedev S.S., Starkov S.O. Autonomous Multi-Camera Neural Network Computer Vision System for Manual Operation Monitoring. Russian Journal of Cybernetics, vol. 5, no. 3, 2024, pp. 24–33 (in Russian) DOI: 10.51790/2712-9942-2024-5-3-03.

5. Хацкелевич А.Н. Применение технологий искусственного интеллекта как один из факторов конкурентоспособности бизнеса в четвертой промышленной революции. Вестник ПНИПУ. Социально-экономические науки, №1, 2024 г., стр. 184–199. DOI: 10.15593/2224-9354/2024.1.14. / Khatskelevich A.N., Rudakov S.A., Egorov G.A. Artificial intelligence technologies as one of the factors of business competitiveness in the fourth industrial revolution. PNRPU Sociology and Economics Bulletin, no. 1, 2024, рp. 184-199 (in Russian) DOI: 10.15593/2224-9354/2024.1.14.

6. Luo X., Li H., Yang X., Yu Y., Cao D. Capturing and understanding workers’ activities in far-field surveillance videos with deep action recognition and Bayesian nonparametric learning. Computer-Aided Civil and Infrastructure Engineering, vol. 34, no. 4, 2019, pp. 333-351. DOI: 10.1111/mice.12419.

7. Варанкин Д.В., Гудошник Е.Э., Мельников С.С. Автоматическая система контроля соблюдения правил охраны труда и производственной безопасности. Бюллетень науки и практики, том 9, №7, 2023, стр. 317-321. DOI: 10.33619/2414-2948/92/44. / Varankin D.V., Gudoshnik E.A., Melnikov S.S. Automatic Health and Safety and Production Safety Monitoring System. Bulletin of Science and Practice, vol. 9, no. 7, 2023, pp. 317-321 (in Russian). DOI: 10.33619/2414-2948/92/44.

8. Shou Z., Wang D., Chang S.-F. Temporal Action Localization in Untrimmed Videos via Multi-stage CNNs. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR’2016), 2016, pp. 1049–1058. DOI: 10.1109/CVPR.2016.119.

9. Liu X., Wang Q., Hu Y., Tang X., Zhang S., Bai S., Bai X. End-to-End Temporal Action Detection With Transformer. IEEE Transactions on Image Processing, vol. 31, 2022, pp. 5427–5441. DOI: 10.1109/TIP.2022.3195321.

10. Штехин С.Е., Карачёв Д.К., Иванова Ю.А. Разработка алгоритма распознавания движений человека методами компьютерного зрения в задаче нормирования рабочего времени. Труды ИСП РАН, том 32, вып. 1, 2020 г., стр. 121-136. DOI: 10.15514/ISPRAS-2020-32(1)-7. / Shtekhin S.E., Karachev D.K., Ivanova Yu.A. Computer vision system for working time estimation by human activities detection in video frames. Trudy ISP RAN/Proc. ISP RAS, vol. 32, issue 1, 2020, pp.121-136 (in Russian) DOI: 10.15514/ISPRAS-2020-32(1)-7.

11. Kadam P., Fang G., Zou J.J. Object Tracking Using Computer Vision: A Review. Computers, 13(6), 2024, 136. DOI: 10.3390/computers13060136.

12. Wang C.-Y., Bochkovskiy A., Liao H.-Y.M. YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR’2023), 2023, pp. 7464-7475. DOI: 10.1109/CVPR52729.2023.00721.

13. Cheng H.K., Schwing A.G. XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model. In Proceedings of the European Conference on Computer Vision (ECCV’2022), 2022, pp. 640–658. DOI: 10.1007/978-3-031-19815-1_37.

14. Kirillov A., Mintun E., Ravi N., Mao H., Rolland C., Gustafson L., Xiao T., Whitehead S., Berg A.C., Lo W.-Y., Dollár P., Girshick R. Segment Anything. In Proceedings of IEEE International Conference on Computer Vision (ICCV’2023), 2023, pp. 4015-4026. DOI: 10.1109/ICCV51070.2023.00371.

15. Zhang Y., Sun P., Jiang Y., Yu D., Weng F., Yuan Z., Luo P., Liu W., Wang X. ByteTrack: Multi-Object Tracking by Associating Every Detection Box. In Proceedings of the European Conference on Computer Vision (ECCV’2022), 2022, pp. 1-21. DOI: 10.1007/978-3-031-20047-2_1

16. Luiten J., Os̆ep A., Dendorfer P., Torr P., Geiger A., Leal-Taixé L., Leibe B. HOTA: A Higher Order Metric for Evaluating Multi-object Tracking. International Journal of Computer Vision, vol. 129, no. 2, 2021, pp. 548-578. DOI: 10.1007/s11263-020-01375-2.

17. Bernardin, K., Stiefelhagen, R.: Evaluating multiple object tracking performance: the clear mot metrics. EURASIP Journal on Image and Video Processing 2008, 1–10 (2008). DOI: 10.1155/2008/246309.

18. Ristani, E., Solera, F., Zou, R., Cucchiara, R., Tomasi, C.: Performance measures and a data set for multi-target, multi-camera tracking. In: European Conference on Computer Vision, pp. 17–35. Springer (2016). DOI: 10.48550/arXiv.1609.01775.

19. Бенчмарк MOT20 для алгоритмов трекинга [Электронный ресурс]. Доступно по адресу: https://motchallenge.net/results/MOT20/ (Дата обращения: 22.12.2025).

20. Список детектируемых инструментов [Электронный ресурс]. Доступно по адресу: https://disk.yandex.ru/i/ZrtRhoUijMdJ6Q (Дата обращения: 22.12.2025).

21. Rombach R., Blattmann A., Lorenz D., Esser P., Ommer B. High-Resolution Image Synthesis with Latent Diffusion Models. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR’2022), 2022, pp. 10674-10685. DOI: 10.1109/CVPR52688.2022.01042.

22. Brooks T., Holynski A., Efros A.A. InstructPix2Pix: Learning to Follow Image Editing Instructions. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR’2023), 2023, pp. 18392-18402. DOI: 10.1109/CVPR52729.2023.01764.

23. Карачёв Д.К., Штехин С.Е., Тарасян В.С., Смолин И.Ю., Исаков М.В. Использование переноса стиля как способ улучшения обобщающей способности нейросети в задаче детекции объектов. Труды ИСП РАН, том 35. вып. 6, 2023 г., стр. 247-264. DOI: 10.15514/ISPRAS-2023-35(6)-16. / Karachev D.K., Shtekhin S.E., Tarasyan V.S., Smolin I.U., Isakov M.V. Style transfer as a way to improve the generalization ability of a neural network in an object detection task. Trudy ISP RAN/Proc. ISP RAS, vol. 35, issue 6, 2023. pp. 247-264. (in Russian) DOI: 10.15514/ISPRAS-2023-35(6)-16.

24. Prokhorenkova L., Gusev G., Vorobev A., Dorogush A.V., Gulin A. CatBoost: unbiased boosting with categorical features. In Proceedings of the 32nd International Conference on Neural Information Processing Systems. Advances in Neural Information Processing Systems (NeurIPS’2018), 2018, pp. 6639–6649.

25. Pham H.H., Khoudour L., Crouzil A., Zegers P., Velastin S.A. Video-based Human Action Recognition using Deep Learning: A Review. arXiv.org. 2022. Available at https://arxiv.org/abs/2208.03775. (accessed 30.06.2025).

26. Carreira J., Zisserman A. Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR’2017), 2017, pp. 4724-4733. DOI: 10.1109/CVPR.2017.502.

27. Duan H., Zhao Y., Chen K., Lin D., Dai B. Revisiting Skeleton-based Action Recognition. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR’2022), 2022, pp. 2959-2968. DOI: 10.1109/CVPR52688.2022.00298.

28. Li Y., Wu C.-Y., Fan H., Mangalam K., Xiong B., Malik J., Feichtenhofer C. MViTv2: Improved Multiscale Vision Transformers for Classification and Detection. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR’2022), 2022, pp. 4794-4804. DOI: 10.1109/CVPR52688.2022.00476.

29. Kriman S., Beliaev S., Ginsburg B., Huang J., Kuchaiev O., Lavrukhin V., Leary R., Li J., Zhang Y. QuartzNet: Deep Automatic Speech Recognition with 1D Time-Channel Separable Convolutions. In Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP’2020), 2020, pp. 6124-6128. DOI: 10.1109/ICASSP40776.2020.9053889.

30. Karpov N., Denisenko A., Minkin F. Golos: Russian Dataset for Speech Research. In Proceedings of Interspeech (Interspeech’2021), 2021, pp. 1419-1423. DOI: 10.21437/Interspeech.2021-46.