A Comparative Study of Training Methods and Architectures of Echo State Networks

This paper examines echo state networks (ESNs), one of the most prevalent approaches to implementing reservoir computing. An ESN consists of a recurrent neural network with fixed (untrained) weights and a readout layer that is typically linear and trainable. This approach enables the creation of energy-efficient and computationally efficient neural networks capable of real-time learning. However, since ESN weights are not trained, their selection constitutes a separate challenge that requires careful analysis. The present paper provides a comparative analysis and review of various ESN architectures and readout layer training methods. This analysis is based on practical experience with implementations and theoretical foundations, including studies of how reservoir dynamics depend on the topology of the connectivity graph and the spectrum of the connectivity matrix. To examine the reservoir structure, tools such as connectivity graph condensation and linearization of dynamics are utilized, along with the introduction of a novel concept termed graph memory. In addition to well-established ESN architectures, the review includes less common or previously unapplied models in the context of reservoir computing, such as reaction-diffusion systems, a single neuron with delay, FORCE learning, and neural fields. Experimental evaluation is conducted through comprehensive experiments on the chaotic Mackey-Glass time series prediction task. This paper not only serves as a practical guide for selecting ESN architectures and readout layer training methods but also identifies promising directions for future research.

1. Jaeger H. The echo state approach to analysing and training recurrent neural networks. German National Research Institute for Computer Science. GMD-Report 148, 2001. Available at: https://www.researchgate.net/publication/215385037_The_echo_state_approach_to_analysing_and_training_recurrent_neural_networks-with_an_erratum_note', accessed 13.03.2026.

2. Maass W., Natschlaeger T., Markram H. Real-time computing without stable states: A new framework for neural computation based on perturbations. Neural Computation, vol. 14, no. 11, pp. 2531-2560, 2002. DOI: 10.1162/089976602760407955.

3. Kirby K., Context dynamics in neural sequential learning. In Proceedings of the Florida artificial intelligence research symposium FLAIRS, 1991, pp. 66-70.

4. Tanaka G. et al., Recent advances in physical reservoir computing: A review. Neural Networks, vol. 115, Mar. 2019, DOI: 10.1016/j.neunet.2019.03.005.

5. Fernando C., Sojakka S. Pattern recognition in a bucket. In Proc. of ECAL, Sep. 2003, pp. 588-597. DOI: 10.1007/978-3-540-39432-7_63.

6. Lukoševičius M. A Practical Guide to Applying Echo State Networks, in Neural Networks: Tricks of the Trade. Reloaded, vol. 7700, Montavon G., Orr G. B., Müller K.-R., Eds., in Lecture notes in computer science, vol. 7700, Springer, 2012, pp. 659-686.

7. Gallicchio C., Micheli A. Deep echo state network (DeepESN): A brief survey. CoRR, vol. abs/1712.04323, 2017, Available at: http://arxiv.org/abs/1712.04323

8. Zhang H., Vargas D. V. A survey on reservoir computing and its interdisciplinary applications beyond traditional machine learning. IEEE Access, vol. 11, pp. 81033-81070, 2023, DOI: 10.1109/access.2023.3299296.

9. Hastie T., Tibshirani R., Friedman J. H. The elements of statistical learning: Data mining, inference, and prediction. In Springer series in statistics. Springer, 2001. Available at: https://books.google.ru/books?id=VRzITwgNV2UC, accessed 13.03.2026.

10. Engl H. W., Hanke M., Neubauer A. Regularization of inverse problems. Kluwer, 1996.

11. Pearson K., On lines and planes of closest fit to systems of points in space. Philosophical Magazine, vol. 2, no. 11, pp. 559-572, 1901, DOI: 10.1080/14786440109462720.

12. Haykin S. S. Adaptive filter theory. Pearson, 2014. Available at: https://books.google.co.za/books?id=J4GRKQEACAAJ, accessed 13.03.2026.

13. Sussillo D., Abbott L. F. Generating coherent patterns of activity from chaotic neural networks. Neuron, vol. 63, no. 4, pp. 544-557, Aug. 2009, DOI: 10.1016/j.neuron.2009.07.018.

14. Steil J. Backpropagation-decorrelation: Online recurrent learning with o(n) complexity. In IEEE International Conference on Neural Networks – Conference Proceedings, Aug. 2004, vol. 2, pp. 843-848. DOI: 10.1109/IJCNN.2004.1380039.

15. Jaeger H. Echo state network. Scholarpedia, 2007, vol. 2, no. 9, p. 2330,

16. DOI: 10.4249/scholarpedia.2330.

17. Manjunath G., Jaeger H. Echo state property linked to an input: Exploring a fundamental characteristic of recurrent neural networks. Neural Computation, vol. 25, no. 3, pp. 671-696, Mar. 2013,

18. DOI: 10.1162/NECO_a_00411.

19. Gallicchio C. Euler State Networks: Non-dissipative Reservoir Computing. arXiv preprint arXiv:2203.09382, 2023, DOI: 10.48550/arXiv.2203.09382.

20. Bollt E. On explaining the surprising success of reservoir computing forecaster of chaos? The universal machine learning dynamical system with contrast to VAR and DMD. Chaos, vol. 31, no. 13108, 2021.

21. Gauthier D. J., Bollt E., Griffith A. Barbosa W. A. S. Next generation reservoir computing. Nature Communications, vol. 12, p. 5564, 2021, DOI: 10.1038/s41467-021-25801-2.

22. Parlitz U. Learning from the past: reservoir computing using delayed variables. Frontiers in Applied Mathematics and Statistics, vol. 10, p. 1221051, Mar. 2024, DOI: 10.3389/fams.2024.1221051.

23. Hart J. D., Larger L., Murphy T. E., Roy R. Delayed dynamical systems: networks, chimeras and reservoir computing. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, vol. 377, no. 2153, 2019, DOI: 10.1098/rsta.2018.0123.

24. Li M. Y., Wei J. Global hopf bifurcation analysis of a neuron network model with time delays. In Infinite dimensional dynamical systems, J. Mallet-Paret, Wu J., Yi Y., Zhu H. (eds.), New York, NY: Springer New York, 2013, pp. 141-168. DOI: 10.1007/978-1-4614-4523-4_5.

25. Coombes S., beim Graben P., Potthast R., Wright J. (eds.), Neural fields: Theory and applications, 1st ed. Springer Berlin, Heidelberg, 2014, pp. X, 487. DOI: 10.1007/978-3-642-54593-1.

26. Turing A. M. The chemical basis of morphogenesis. Philosophical Transactions of the Royal Society of London. B, Biological Sciences, vol. 237, no. 641, pp. 37-72, Aug. 1952, DOI: 10.1098/rstb.1952.0012.

27. Amari S. Homogeneous nets of neuron-like elements. Biological Cybernetics, vol. 17, pp. 211-220, 1975.

28. Amari S. Dynamics of pattern formation in lateral-inhibition type neural fields. Biological Cybernetics, vol. 27, pp. 77-87, 1977.

29. Nunez P. L. The brain wave equation: A model for the EEG. Mathematical Biosciences, vol. 21, no. 3, pp. 279–297, 1974.

30. Cook B. J., Peterson A. D. H., Woldman W., Terry J. R. Neural field models: A mathematical overview and unifying framework. Mathematical Neuroscience and Applications, vol. 2, Mar. 2022,

31. DOI: 10.46298/mna.7284.

32. Gorecki J. et al. Chemical computing with reaction–diffusion processes. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, vol. 373, no. 20140219, 2015, DOI: 10.1098/rsta.2014.0219.

33. Mackey M. C., Glass L. Oscillation and chaos in physiological control systems. Science, vol. 197, no. 4300, pp. 287-289, 1977, DOI: 10.1126/science.267326.

34. Jaeger H. The "echo state" approach to analysing and training recurrent neural networks-with an erratum note. Bonn, Germany: German National Research Center for Information Technology. GMD Technical Report, vol. 148, Jan. 2001.

35. Wringe C., Trefzer M., Stepney S. Reservoir computing benchmarks: A tutorial review and critique. International Journal of Parallel, Emergent and Distributed Systems, vol. 40, no. 4, pp. 313-351, Mar. 2025, DOI: 10.1080/17445760.2025.2472211.

36. Ozaki Y., Watanabe S., Yanase T. OptunaHub: A platform for black-box optimization. arXiv preprint arXiv:2510.02798, 2025. Available at: https://arxiv.org/pdf/2510.02798, accessed 12.03.2026.