Обзор методов контроля перегрузки с использованием машинного обучения

    Контроль перегрузки является ключевым аспектом современных сетей. Первые алгоритмы контроля перегрузки, такие как TCP Tahoe и TCP Reno, были разработаны в конце XX века, и их основные идеи остаются актуальными до сих пор. С развитием высокоскоростных сетей для них были созданы специализированные алгоритмы, например, TCP BIC и TCP CUBIC. Однако классические алгоритмы, основанные на определённых правилах, не всегда оказываются эффективными во всех сетевых условиях, и с развитием 4G, 5G и спутниковой связи задача контроля перегрузки стала более актуальной. Это привело к появлению решений этой задачи на основе машинного обучения и обучения с подкреплением, в частности таких, которые способны адаптироваться к динамически изменяющимся условиям сети. В статье представлены и рассмотрены как классические алгоритмы контроля перегрузки, так и наиболее популярные и новые алгоритмы, основанные на машинном обучении, а также некоторые реализации с использованием технологии multipath. Кроме того, выделены наиболее значимые проблемы алгоритмов на основе машинного обучения и обсуждены потенциальные направления будущих исследований в данной области.

1. Jiang H. et al. When machine learning meets congestion control: A survey and comparison //Computer Networks. – 2021. – Т. 192. – С. 108033.

2. Wei W., Gu H., Li B. Congestion control: A renaissance with machine learning //IEEE network. – 2021. – Т. 35. – №. 4. – С. 262-269.

3. Jacobson V. Congestion avoidance and control //ACM SIGCOMM computer communication review. – 1988. – Т. 18. – №. 4. – С. 314-329.

4. Mo J. et al. Analysis and comparison of TCP Reno and Vegas //IEEE INFOCOM'99. Conference on Computer Communications. Proceedings. Eighteenth Annual Joint Conference of the IEEE Computer and Communications Societies. The Future is Now (Cat. No. 99CH36320). – IEEE, 1999. – Т. 3. – С. 1556-1563.

5. Floyd S., Henderson T. The NewReno modification to TCP's fast recovery algorithm. – 1999. – №. rfc2582.

6. Kelly T. Scalable TCP: Improving performance in highspeed wide area networks //ACM SIGCOMM computer communication Review. – 2003. – Т. 33. – №. 2. – С. 83-91.

7. Floyd S. HighSpeed TCP for large congestion windows. – 2003. – №. rfc3649.

8. Xu L., Harfoush K., Rhee I. Binary increase congestion control (BIC) for fast long-distance networks //IEEE INFOCOM 2004. – IEEE, 2004. – Т. 4. – С. 2514-2524.

9. Ha S., Rhee I., Xu L. CUBIC: a new TCP-friendly high-speed TCP variant //ACM SIGOPS operating systems review. – 2008. – Т. 42. – №. 5. – С. 64-74.

10. Brakmo L. S., O'malley S. W., Peterson L. L. TCP Vegas: New techniques for congestion detection and avoidance //Proceedings of the conference on Communications architectures, protocols and applications. – 1994. – С. 24-35.

11. Srijith K. N., Jacob L., Ananda A. L. TCP Vegas-A: Solving the fairness and rerouting issues of TCP Vegas //Conference Proceedings of the 2003 IEEE International Performance, Computing, and Communications Conference, 2003. – IEEE, 2003. – С. 309-316.

12. Srijith K. N., Jacob L., Ananda A. L. TCP Vegas-A: Improving the performance of TCP Vegas //Computer communications. – 2005. – Т. 28. – №. 4. – С. 429-440.

13. Fu C. P., Liew S. C. TCP Veno: TCP enhancement for transmission over wireless access networks //IEEE Journal on selected areas in communications. – 2003. – Т. 21. – №. 2. – С. 216-228.

14. Mascolo S. et al. TCP Westwood: Bandwidth estimation for enhanced transport over wireless links //Proceedings of the 7th annual international conference on Mobile computing and networking. – 2001. – С. 287-297.

15. Cardwell N. et al. Bbr: Congestion-based congestion control: Measuring bottleneck bandwidth and round-trip propagation time //Queue. – 2016. – Т. 14. – №. 5. – С. 20-53.

16. Cardwell N. et al. BBRv2: A model-based congestion control performance optimization //Proc. IETF 106th Meeting. – 2019. – С. 1-32.

17. Zhang Y., Cui L., Tso F. P. Modest BBR: Enabling better fairness for BBR congestion control //2018 IEEE symposium on computers and communications (ISCC). – IEEE, 2018. – С. 00646-00651.

18. Arun V., Balakrishnan H. Copa: Practical {Delay-Based} congestion control for the internet //15th USENIX Symposium on Networked Systems Design and Implementation (NSDI 18). – 2018. – С. 329- 342.

19. Watkins C. J. C. H., Dayan P. Q-learning //Machine learning. – 1992. – Т. 8. – С. 279-292

20. Gu S. et al. Continuous deep q-learning with model-based acceleration //International conference on machine learning. – PMLR, 2016. – С. 2829-2838.

21. Konda V., Tsitsiklis J. Actor-critic algorithms //Advances in neural information processing systems. – 1999. – Т. 12.

22. Labao A. B., Martija M. A. M., Naval P. C. A3C-GS: Adaptive moment gradient sharing with locks for asynchronous actor–critic agents //IEEE Transactions on Neural Networks and Learning Systems. – 2020. – Т. 32. – №. 3. – С. 1162-1176.

23. Schulman J. et al. Proximal policy optimization algorithms //arXiv preprint arXiv:1707.06347. – 2017.

24. El Khayat I., Geurts P., Leduc G. Improving TCP in wireless networks with an adaptive machine-learnt classifier of packet loss causes //NETWORKING 2005. Networking Technologies, Services, and Protocols; Performance of Computer and Communication Networks; Mobile and Wireless Communications Systems: 4th International IFIP-TC6 Networking Conference, Waterloo, Canada, May 2-6, 2005. Proceedings 4. – Springer Berlin Heidelberg, 2005. – С. 549-560.

25. Geurts P., El Khayat I., Leduc G. A machine learning approach to improve congestion control over wireless computer networks //Fourth IEEE International Conference on Data Mining (ICDM'04). – IEEE, 2004. – С. 383-386.

26. Jayaraj A., Venkatesh T., Murthy C. S. R. Loss classification in optical burst switching networks using machine learning techniques: improving the performance of tcp //IEEE Journal on Selected Areas in Communications. – 2008. – Т. 26. – №. 6. – С. 45-54.

27. Arouche Nunes B. A. et al. A machine learning framework for TCP round-trip time estimation //EURASIP Journal on Wireless Communications and Networking. – 2014. – Т. 2014. – С. 1-22.

28. Dai T., Zhang X., Guo Z. Learning-based congestion control for internet video communication over wireless networks //2018 IEEE International Symposium on Circuits and Systems (ISCAS). – IEEE, 2018. – С. 1-5.

29. Parlos A. G. Identification of the internet end-to-end delay dynamics using multi-step neuro-predictors //Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No. 02CH37290). – IEEE, 2002. – Т. 3. – С. 2460-2465.

30. Bui V. et al. Long horizon end-to-end delay forecasts: A multi-step-ahead hybrid approach //2007 12th IEEE Symposium on Computers and Communications. – IEEE, 2007. – С. 825-832.

31. Belhaj S., Tagina M. Modeling and Prediction of the Internet End-to-end Delay using Recurrent Neural Networks //J. Networks. – 2009. – Т. 4. – №. 6. – С. 528-535.

32. Ahmed A. H. et al. Predicting high delays in mobile broadband networks //IEEE Access. – 2021. – Т. 9. – С. 168999-169013.

33. Li W. et al. QTCP: Adaptive congestion control with reinforcement learning //IEEE Transactions on Network Science and Engineering. – 2018. – Т. 6. – №. 3. – С. 445-458.

34. Jay N. et al. A deep reinforcement learning perspective on internet congestion control //International Conference on Machine Learning. – PMLR, 2019. – С. 3050-3059.

35. Fang J. et al. Reinforcement learning for bandwidth estimation and congestion control in real-time communications //arXiv preprint arXiv:1912.02222. – 2019.

36. Abbasloo S., Yen C. Y., Chao H. J. Classic meets modern: A pragmatic learning-based congestion control for the internet //Proceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication. – 2020. – С. 632-647.

37. Sivakumar V. et al. Mvfst-rl: An asynchronous rl framework for congestion control with delayed actions //arXiv preprint arXiv:1910.04054. – 2019.

38. Zhang Z. et al. PBQ-Enhanced QUIC: QUIC with Deep Reinforcement Learning Congestion Control Mechanism //Entropy. – 2023. – Т. 25. – №. 2. – С. 294.

39. Tian H. et al. Spine: An efficient DRL-based congestion control with ultra-low overhead //Proceedings of the 18th International Conference on emerging Networking Experiments and Technologies. – 2022. – С. 261-275.

40. Shi H., Wang J. Intelligent TCP congestion control policy optimization //Applied Sciences. – 2023. – Т. 13. – №. 11. – С. 6644.

41. Andrade-Zambrano A. R. et al. A Reinforcement Learning Congestion Control Algorithm for Smart Grid Networks //IEEE Access. – 2024.

42. Wischik D. et al. Design, implementation and evaluation of congestion control for multipath {TCP} //8th USENIX Symposium on Networked Systems Design and Implementation (NSDI 11). – 2011.

43. Nguyen K. et al. An approach to reinforce multipath TCP with path-aware information //Sensors. – 2019. – Т. 19. – №. 3. – С. 476.

44. Chao L. et al. A brief review of multipath tcp for vehicular networks //Sensors. – 2021. – Т. 21. – №. 8. – С. 2793.

45. Deng S. et al. WiFi, LTE, or both? Measuring multi-homed wireless internet performance //Proceedings of the 2014 Conference on Internet Measurement Conference. – 2014. – С. 181-194.

46. Scharf M., Kiesel S. NXG03-5: Head-of-line Blocking in TCP and SCTP: Analysis and Measurements //IEEE Globecom 2006. – IEEE, 2006. – С. 1-5.

47. De Coninck Q., Bonaventure O. Multipath quic: Design and evaluation //Proceedings of the 13th international conference on emerging networking experiments and technologies. – 2017. – С. 160-166.

48. Viernickel T. et al. Multipath QUIC: A deployable multipath transport protocol //2018 IEEE International Conference on Communications (ICC). – IEEE, 2018. – С. 1-7.

49. Wang S. et al. Deep reinforcement learning for dynamic multichannel access //International Conference on Computing, Networking and Communications (ICNC). – 2017. – С. 257-265.

50. Lee S., Yoo J. Reinforcement learning based multipath QUIC scheduler for multimedia streaming //Sensors. – 2022. – Т. 22. – №. 17. – С. 6333.

51. Luo J., Su X., Liu B. A reinforcement learning approach for multipath TCP data scheduling //2019 IEEE 9th Annual Computing and Communication Workshop and Conference (CCWC). – IEEE, 2019. – С. 0276-0280.

52. Roselló M. M. Multi-path scheduling with deep reinforcement learning //2019 European Conference on Networks and Communications (EuCNC). – IEEE, 2019. – С. 400-405.

53. Yu C. et al. Reliable cybertwin-driven concurrent multipath transfer with deep reinforcement learning //IEEE Internet of Things Journal. – 2021. – Т. 8. – №. 22. – С. 16207-16218.

54. Deutschmann J., Hielscher K. S. J., German R. An ns-3 model for multipath communication with terrestrial and satellite links //Measurement, Modelling and Evaluation of Computing Systems: 20th International GI/ITG Conference, MMB 2020, Saarbrücken, Germany, March 16–18, 2020, Proceedings 20. – Springer International Publishing, 2020. – С. 65-81.

55. Mai T. et al. Self-learning congestion control of MPTCP in satellites communications //2019 15th International Wireless Communications & Mobile Computing Conference (IWCMC). – IEEE, 2019. – С. 775-780.

56. Xing Z. et al. A multipath routing algorithm for satellite networks based on service demand and traffic awareness //Frontiers of Information Technology & Electronic Engineering. – 2023. – Т. 24. – №. 6. – С. 844-858.

57. Lv G. et al. Chorus: Coordinating Mobile Multipath Scheduling and Adaptive Video Streaming //Proceedings of the 30th Annual International Conference on Mobile Computing and Networking. – 2024. – С. 246-262.

58. Berenji H. R. Fuzzy Q-learning: a new approach for fuzzy dynamic programming //Proceedings of 1994 IEEE 3rd International Fuzzy Systems Conference. – IEEE, 1994. – С. 486-491.

59. Zhou W. et al. TCP Vegas-V: Improving the performance of TCP Vegas //International Conference on Automatic Control and Artificial Intelligence (ACAI 2012). – IET, 2012. – С. 2034-2039.

60. Dong M. et al. {PCC} vivace: {Online-Learning} congestion control //15th USENIX Symposium on Networked Systems Design and Implementation (NSDI 18). – 2018. – С. 343-356.

61. Dong M. et al. {PCC}: Re-architecting congestion control for consistent high performance //12th USENIX Symposium on Networked Systems Design and Implementation (NSDI 15). – 2015. – С. 395-408.

62. De Oliveira R. L. S. et al. Using mininet for emulation and prototyping software-defined networks //2014 IEEE Colombian conference on communications and computing (COLCOM). – Ieee, 2014. – С. 1-6.

63. Riley G. F., Henderson T. R. The ns-3 network simulator //Modeling and tools for network simulation. – Berlin, Heidelberg: Springer Berlin Heidelberg, 2010. – С. 15-34.

64. Beale J., Orebaugh A., Ramirez G. Wireshark & Ethereal network protocol analyzer toolkit. – Elsevier, 2006.

65. Deri L. et al. ndpi: Open-source high-speed deep packet inspection //2014 International Wireless Communications and Mobile Computing Conference (IWCMC). – IEEE, 2014. – С. 617-622.

66. Mishra A., Leong B. Containing the Cambrian Explosion in QUIC Congestion Control //Proceedings of the 2023 ACM on Internet Measurement Conference. – 2023. – С. 526-539.