A Decade of Advancements in Program Synthesis from Natural Language: A Systematic Literature Review
https://doi.org/10.15514/ISPRAS-2024-36(6)-4
Abstract
Program Synthesis is the process of automatically generating software from a requirement specification. This paper presents a systematic literature review focused on program synthesis from specifications expressed in natural language. The research problem centers on the complexity of automatically generating accurate and robust code from high-level, ambiguous natural language descriptions – a barrier that limits the broader adoption of automatic code generation in software development. To address this issue, the study systematically examines research published between 2014 and 2024, focusing on works that explore various approaches to program synthesis from natural language inputs. The review follows a rigorous methodology, incorporating search strings tailored to capture relevant studies from five major data sources: IEEE, ACM, Springer, Elsevier, and MDPI. The selection process applied strict inclusion and exclusion criteria, resulting in a final set of 20 high-quality studies. The findings reveal significant advancements in the field, particularly in the integration of large language models (LLMs) with program synthesis techniques. The review also highlights the challenges and concludes by outlining key trends and proposing future research directions aimed at overcoming these challenges and expanding the applicability of program synthesis across various domains.
Supporting agencies: This work was partially developed under the support of the National Council of Humanities, Science and Technology (CONAHCYT-Mexico) in the scope of the project “Infraestructura para Agilizar el Desarrollo de Sistemas Centrados en el Usuario” (Catedras, Ref. 3053). In addition, the authors thank CONAHCYT for the doctoral scholarship granted to the first author. We also thank the Universidad Veracruzana for the support in the development of this research.
About the Authors
Rolando RAMÍREZ-RUEDA
Facultad de Estadistica e Informatica, Universidad Veracruzana
Mexico
Mexico
PhD Student in Computer Science from the University of Veracruz in Mexico. Professor in the systems department of Veracruz University Institute in Mexico. Research interests: Automated Reasoning, Artificial Intelligence, Multiagent Systems.
Edgard BENÍTEZ-GUERRERO
Facultad de Estadistica e Informatica, Universidad Veracruzana
Mexico
Mexico
PhD in Computer Science from the University of Grenoble in France. Professor at the Faculty of Statistics and Informatics of the University of Veracruz in Mexico. Research interests: Human Computer Interaction, Artificial Intelligence, Collaborative Computing, Data Management and Visualization.
Carmen MEZURA-GODOY
Facultad de Estadistica e Informatica, Universidad Veracruzana
Mexico
Mexico
PhD in Computer Science from the University of Savoie in France. Professor at the Faculty of Statistics and Informatics of the University of Veracruz in Mexico. Main research interests: Human-Computer Interaction, User eXperience-UX, Computer-Supported Collaborative Work, Visualization and Multiagent Systems.
Everardo BARCENAS
Universidad Nacional Autonoma de Mexico Ciudad de Mexico
Mexico
Mexico
Holds a PhD in Computing Science from the University of Grenoble, and is an Assistant Professor in the Computing Engineering Department of the National University of Mexico. His research interests include: Modal Logics, Proof Theory, Automated Reasoning, Description Logics, Model Checking and Formal Verification.
References
1. J. Fu, F. B. Bastani, and I.-L. Yen, in Innovations for Requirement Analysis. From Stakeholders’ Needs to Formal Designs, edited by B. Paech and C. Martell (Springer Berlin Heidelberg, Berlin, Heidelberg, 2007), pp. 43–61, ISBN 978-3-540-89778-1.
2. Z. Manna and R. J. Waldinger, Toward automatic program synthesis (1971), URL https://doi.org/10.1145/362566. 362568.
3. S. Jiho and N. Jaechang, A survey of automatic code generation from natural language (2021).
4. A. Bandi, P. V. S. R. Adapa, and Y. E. V. P. K. Kuchi, The power of generative ai: A review of requirements, models, input–output formats, evaluation metrics, and challenges (2023), URL https://www.mdpi.com/1999-5903/15/8/260.
5. S. Gulwani, in Formal Methods in Computer Aided Design (2010), pp. 1–1.
6. Z. Manna and R. Waldinger, A deductive approach to program synthesis (1980), URL https://doi.org/10.1145/357084. 357090.
7. E. Kitzelmann, in Approaches and Applications of Inductive Programming, edited by U. Schmid, E. Kitzelmann, and R. Plasmeijer (Springer Berlin Heidelberg, Berlin, Heidelberg, 2010), pp. 50–73, ISBN 978-3-642-11931-6.
8. A. F. Subahi, Cognification of program synthesis—a systematic feature-oriented analysis and future direction (2020), URL https://www.mdpi.com/2073-431X/9/2/27.
9. J. Sauvola, S. Tarkoma, M. Klemettinen, J. Riekki, and D. Doermann, Future of software development with generative ai (2024).
10. C. Ebert and P. Louridas, Generative ai for software practitioners (2023).
11. K. S. Kaswan, J. S. Dhatterwal, K. Malik, and A. Baliyan, in 2023 International Conference on Communication, Security and Artificial Intelligence (ICCSAI) (2023), pp. 699–704.
12. G. Brunner, A. Konrad, Y. Wang, and R. Wattenhofer, Midi-vae: Modeling dynamics and instrumentation of music with applications to style transfer (2018), URL https://api.semanticscholar.org/CorpusID:49317433.
13. T. Karras, S. Laine, and T. Aila, A style-based generator architecture for generative adversarial networks (2021), URL https://doi.org/10.1109/TPAMI.2020.2970919.
14. D. Jonassen, J. Howland, J. L. Moore, and R. Marra, in Learning to Solve Problems with Technology: A Constructivist Perspective (2002).
15. E. D. Liddy, Natural language processing. (2001).
16. D. Jatnika, M. A. Bijaksana, and A. A. Suryani, Word2vec model analysis for semantic similarities in english words (2019), the 4th International Conference on Computer Science and Computational Intelligence (ICCSCI 2019) : Enabling Collaboration to Escalate Impact of Research Results for Society, URL https://www.sciencedirect.com/science/article/ pii/S1877050919310713.
17. Venkatesh, S. U. Hegde, Z. A. S, and N. Y, in 2021 International Conference on Advances in Electrical, Computing, Communication and Sustainable Technologies (ICAECT) (2021), pp. 1–8.
18. L. Wang, Y. Ling, Z. Yuan, M. Shridhar, C. Bao, Y. Qin, B. Wang, H. Xu, and X. Wang, Gensim: Generating robotic simulation tasks via large language models (2024), 2310.01361.
19. A. Namboori, S. Mangale, A. Rosenbaum, and S. Soltan, Gemquad : Generating multilingual question answering datasets from large language models using few shot learning (2024), 2404.09163.
20. T. Mikolov, M. Karafiat, L. Burget, J. Cernocky´, and S. Khudanpur, in Recurrent neural network based language model (2010), vol. 2, pp. 1045–1048.
21. T. Mikolov, A. Deoras, D. Povey, L. Burget, and J. Cernocky, Strategies for training large scale neural network language models (2011).
22. S. Islam, H. Elmekki, A. Elsebai, J. Bentahar, N. Drawel, G. Rjoub, and W. Pedrycz, A comprehensive survey on applications of transformers for deep learning tasks (2023), 2306.07303.
23. P. Yin and G. Neubig, in A Syntactic Neural Model for General-Purpose Code Generation (2017), pp. 440–450.
24. B. Kitchenham, O. Pearl Brereton, D. Budgen, M. Turner, J. Bailey, and S. Linkman, Information and Software Technology 51, 7 (2009), ISSN 0950-5849, special Section – Most Cited Articles in 2002 and Regular Research Papers, URL http: //www.sciencedirect.com/science/article/pii/S0950584908001390.
25. K. Petersen, R. Feldt, S. Mujtaba, and M. Mattsson, in Proceedings of the 12th International Conference on Evaluation and Assessment in Software Engineering (BCS Learning and Development Ltd., 2008), EASE’08, p. 68–77.
26. N. S. M. Yusop, J. Grundy, and R. Vasa, IEEE Transactions on Software Engineering 43, 848 (2017).
27. R. Moguel-S´anchez, C. Mart´ınez-Palacios, J. Ochar´an-Hern´andez, X. Lim´on, and A. S´anchez-Garc´ıa, Programming and Computer Software 49, 712 (2024).
28. P. O. Silva-Vasquez, V. Y. Rosales-Morales, and E. Ben´ıtez-Guerrero, Program. Comput. Softw. 48, 685–701 (2022), ISSN 0361-7688, URL https://doi.org/10.1134/S0361768822080187.
29. U. Dissemination, The database of abstracts of reviews of effects (dare) (2002).
30. P. Archana, P. B. Harish, N. Rajan, S. P, and N. S. Kumar, in 2021 Asian Conference on Innovation in Technology (ASIANCON) (2021), pp. 1–8.
31. D. Wang, W. Dong, and Y. Zhang, in 2020 35th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) (2020), pp. 98–104.
32. H. Phan, A. Sharma, and A. Jannesari, in 2021 36th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) (2021), pp. 219–226.
33. Z. Nan, H. Guan, and X. Shen, HISyn: Human Learning-Inspired Natural Language Programming (Association for Computing Machinery, New York, NY, USA, 2020), p. 75–86, ISBN 9781450370431, URL https://doi.org/10.1145/3368089. 3409673.
34. Z. Zhang, S. Wu, R. Jiang, M. Pan, and T. Zhang, in Proceedings of the Tenth Asia-Pacific Symposium on Internetware (Association for Computing Machinery, New York, NY, USA, 2018), Internetware ’18, ISBN 9781450365901, URL https: //doi.org/10.1145/3275219.3275229.
35. Z. Nan, H. Guan, X. Shen, and C. Liao, in Proceedings of the 30th ACM SIGPLAN International Conference on Compiler Construction (Association for Computing Machinery, New York, NY, USA, 2021), CC 2021, p. 141–152, ISBN 9781450383257, URL https://doi.org/10.1145/3446804.3446852.
36. I. Gavran, E. Darulova, and R. Majumdar, Interactive synthesis of temporal specifications from examples and natural language (2020), URL https://doi.org/10.1145/3428269.
37. S. Zamanirad, B. Benatallah, M. C. Barukh, F. Casati, and C. Rodriguez, in 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE) (2017), pp. 832–837.
38. H. Guan, X. Shen, and H. Krim, in SC17: International Conference for High Performance Computing, Networking, Storage and Analysis (2017), pp. 1–14.
39. T. Gvero and V. Kuncak, in 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering (2015), vol. 2, pp. 689–692.
40. N. Jain, S. Vaidyanath, A. Iyer, N. Natarajan, S. Parthasarathy, S. Rajamani, and R. Sharma, in 2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE) (2022), pp. 1219–1231.
41. N. Tao, A. Ventresque, and T. Saber, in 2023 IEEE Latin American Conference on Computational Intelligence (LA-CCI) (2023), pp. 1–6.
42. S. Lee, S. Y. Nam, and J. Kim, Program synthesis through learning the input-output behavior of commands (2022).
43. N. Tao, A. Ventresque, and T. Saber, in Optimization and Learning – 5th International Conference, OLA 2022, Syracuse, Sicilia, Italy, July 18-20, 2022, Proceedings, edited by B. Dorronsoro, M. Pavone, A. Nakib, and E.-G. Talbi (Springer, 2022), vol. 1684 of Communications in Computer and Information Science, pp. 240–252, ISBN 978-3-031-22039-5, URL https://doi.org/10.1007/978-3-031-22039-5_19.
44. A. Bhardwaj, P. Khanna, S. Kumar, and Pragya, Generative model for nlp applications based on component extraction (2020), international Conference on Computational Intelligence and Data Science, URL https://www.sciencedirect. com/science/article/pii/S1877050920308577.
45. K. Rahmani, M. Raza, S. Gulwani, V. Le, D. Morris, A. Radhakrishna, G. Soares, and A. Tiwari, Multi-modal program inference: a marriage of pre-trained language models and component-based synthesis (2021), URL https://doi.org/10. 1145/3485535.
46. P. Denny, J. Leinonen, J. Prather, A. Luxton-Reilly, T. Amarouche, B. A. Becker, and B. N. Reeves, in Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 (Association for Computing Machinery, New York, NY, USA, 2024), SIGCSE 2024, p. 296–302, ISBN 9798400704239, URL https://doi.org/10.1145/3626252.3630909.
47. M. Kacmajor and J. D. Kelleher, Automatic acquisition of annotated training corpora for test-code generation (2019), URL https://www.mdpi.com/2078-2489/10/2/66.
48. M.-F. Wong, S. Guo, C.-N. Hang, S.-W. Ho, and C.-W. Tan, Natural language generation and understanding of big code for ai-assisted programming: A review (2023), URL https://www.mdpi.com/1099-4300/25/6/888.
49. C. E. A. Coello, M. N. Alimam, and R. Kouatly, Effectiveness of chatgpt in coding: A comparative analysis of popular large language models (2024), URL https://www.mdpi.com/2673-6470/4/1/5.
50. M. Rabinovich, M. Stern, and D. Klein, Abstract syntax networks for code generation and semantic parsing (2017), URL https://arxiv.org/abs/1704.07535.
Review
For citations:
RAMÍREZ-RUEDA R., BENÍTEZ-GUERRERO E., MEZURA-GODOY C., BARCENAS E. A Decade of Advancements in Program Synthesis from Natural Language: A Systematic Literature Review. Proceedings of the Institute for System Programming of the RAS (Proceedings of ISP RAS). 2024;36(6):59-82. https://doi.org/10.15514/ISPRAS-2024-36(6)-4
Email this article 