Оценка корректности сгенерированного нейросетями кода: вероятностный подход
https://doi.org/10.15514/ISPRAS-2026-38(2)-8
Аннотация
Большие языковые модели находят всё более широкое применение в разработке программного обеспечения. Однако исследование корректности генерируемого ими кода осложняется недостаточной формализацией понятия корректности программ. В данной работе описан вероятностный подход к оценке корректности кода, генерируемого нейросетями. Предложена метрика корректности TSA (Test Suite Accuracy), естественным образом выводимая в данной формализации, а также проводится сравнение с метрикой Pass@1. Проведённые эксперименты с 5 языковыми моделями Phi-1, Phi-2, Phi-3-mini-4k, Phi-4-mini и Qwen2.5-Coder подтверждают описанные теоретические свойства метрик. Практическим результатом проведённого исследования являются набор задач HumanEval++, расширяющий набор данных HumanEval+, и построенная на его основе реализация метрики TSA.
Ключевые слова
большие языковые модели,
программная инженерия,
генерация кода,
качество кода,
корректность кода,
метрики.
Об авторе
Давид Арменович АВАГЯН
Московский государственный университет имени М.В. Ломоносова
Россия
Россия
Аспирант кафедры алгоритмических языков факультета вычислительной математики и кибернетики Московского государственного университета имени М.В. Ломоносова. Сфера научных интересов: нейросетевая генерация кода, метрики качества программ.
Список литературы
1. IEEE Computer Society. IEEE Std 610.12 1990. IEEE standard glossary of software engineering terminology, 1991, pp. 1 84.
2. Hou X., Zhao Y., Liu Y., Yang Z., Wang K., Li Li, Luo X., Lo D., Grundy J., Wang H. Large Language Models for Software Engineering: A Systematic Literature Review. arXiv preprint arXiv:2308.10620, 2023.
3. Hey T., Keim J., Koziolek A., Tichy W. F. NoRBERT. Transfer Learning for Requirements Classification. In 2020 IEEE 28th International Requirements Engineering Conference (RE), IEEE 2020, pp. 169 179.
4. Ma L., Liu S., Li Y., Xie X., Bu L. SpecGen: Automated Generation of Formal Program Specifications via Large Language Models. arXiv preprint arXiv:2401.08807, 2024.
5. Mandal S., Chethan A., Janfaza V., Mahmud S. M., Anderson T. A., Turek J., Tithi J. J., Muzahid A. Large Language Models Based Automatic Synthesis of Software Specifications. arXiv preprint arXiv:2304.09181, 2023.
6. Cassano F., Gouwar J., Nguyen D., Nguyen S., Phipps-Costin L., Pinckney D., Yee M. H., Anderson C. J., Feldman M. Q. et al. MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation. In 2023 IEEE Transactions on Software Engineering, 49(7), IEEE 2023, pp. 3675 3691.
7. Chen F., Fard F. H., Lo D., Bryksin T. On the Transferability of Pre trained Language Models for Low resource Programming Languages. In 2022 IEEE/ACM 30th International Conference on Program Comprehension (ICPC), IEEE 2022, pp. 401 412.
8. Fan G., Chen S., Gao C., Xiao J., Zhang T., Feng Z. Rapid: Zero shot Domain Adaptation for Code Search with Pre trained Models. In 2024 ACM Transactions on Software Engineering and Methodology, ACM 2024, 33(5), pp. 1 35.
9. Schäfer M., Nadi S., Eghbali A., Tip F. An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation. In 2023 IEEE Transactions on Software Engineering, IEEE 2023, 50(1), pp. 85 105.
10. Steenhoek B., Gao H., Le W. Dataflow Analysis Inspired Deep Learning for Efficient Vulnerability Detection. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, IEEE 2024, pp. 1 13.
11. Li Y., Ren Z., Wang Z., Yang L., Dong L., Zhong C., Zhang H. Fine SE: Integrating Semantic Features and Expert Features for Software Effort Estimation. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, IEEE 2024, pp. 1 12.
12. Nappa A., Johnson R., Bilge L. et al. The Attack of the Clones: A Study of the Impact of Shared Code on Vulnerability Patching. In 2015 IEEE symposium on security and privacy, IEEE 2015, pp. 692 708.
13. Ardito L., Coppola R., Barbato L., Verga D. A Tool Based Perspective on Software Code Maintainability Metrics: A Systematic Literature Review. Scientific Programming 2020, pp. 1 26.
14. Buse R. P. L., Weimer W. R. Learning a Metric for Code Reliability. IEEE Transactions on Software Engineering, IEEE 2009, 36(4), pp. 546 558.
15. Peitek N., Apel S., Parmin C. et al. Program Comprehension and Code Complexity Metrics: An fMRI Study. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), IEEE 2021, pp. 524 536.
16. Markovtsev V., Long W., Mougard H. et al. STYLE ANALYZER: fixing code style inconsistencies with interpretable unsupervised algorithms. In 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR), IEEE 2019, pp. 468 478.
17. Raemaekers S., Van Deursen A., Visser J. Measuring software library stability through historical version analysis. In 2012 28th IEEE International Conference on Software Maintenance (ICSM), IEEE 2012, pp. 378 387.
18. Zheng J., Cao B., Ma Z., Pan R., Lin H., Lu Y., Han X., Sun L. Beyond Correctness: Benchmarking Multi dimensional Code Generation for Large Language Models. arXiv preprint arXiv:2407.11470, 2024.
19. Chen L., Guo Q., Jia H., Zeng Z., Wang X., Xu Y. et al. A Survey on Evaluating Large Language Models in Code Generation Tasks. arXiv preprint arXiv:2408.16498, 2024.
20. Elnashar A., Moundas M., Schmidt D. C., Spencer Smith J., White J. Evaluating the Performance of LLM Generated Code for ChatGPT4 and AutoGen Along with Top Rated Human Solutions. In 2024 19th International Conference on Software Technologies (ICSOFT), 2024, pp. 258 270.
21. Jiang J., Wang F., Shen J., Kim S., Kim S. A Survey on Large Language Models for Code Generation. arXiv preprint arXiv:2406.00515, 2024.
22. Yetiştiren B., Özsoy I., Ayerdem M., Tüzün E. Evaluating the Code Quality of AI assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT. arXiv preprint arXiv:2304.10778, 2023.
23. Huang D., Qing Y., Shang W., Cui H., Zhang J. EffiBench: Benchmarking the Efficiency of Automatically Generated Code. In: Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., Zhang, C. (eds.) Advances in Neural Information Processing Systems, Curran Associates, Inc., 2024, 37, pp. 11506 11544.
24. Peng Y., Wan J., Li Y., Ren X. COFFE: A Code Efficiency Benchmark for Code Generation. In ACM International Conference on the Foundations of Software Engineering (FSE), ACM 2025.
25. Niu C., Zhang T., Li C., Luo B., Ng V. On Evaluating the Efficiency of Source Code Generated by LLMs. In Proceedings of the 2024 IEEE/ACM 1st International Conference on AI Foundation Models and Software Engineering (FORGE), ACM 2024, pp. 103 107.
26. Coignion T., Quinton C., Rouvoy R. A Performance Study of LLM Generated Code on Leetcode. In Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering (EASE), ACM 2024, pp. 79 89.
27. Qiu R., Zeng W. W., Ezick J., Lott C., Tong H. How Efficient is LLM Generated Code? A Rigorous & High Standard Benchmark. 13th International Conference on Learning Representations (ICLR). (2025)
28. Liu J., Xie S., Wang J., Wei Y., Ding Y., Zhang L. Evaluating Language Models for Efficient Code Generation. First Conference on Language Modeling, 2024.
29. Du M., Luu A.T., Ji B., Liu Q., Ng S.K. Mercury: A Code Efficiency Benchmark for Code Large Language Models. In Globerson A., Mackey L., Belgrave D., Fan A., Paquet U., Tomczak J., Zhang C. (eds.) Advances in Neural Information Processing Systems, Curran Associates, Inc. 2024, vol. 37, pp. 16601 16622.
30. Chen M., Tworek J., Jun H. et al. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374, 2021.
31. Li Y., Choi D., Chung J. et al. Competition Level Code Generation with AlphaCode. Science, 2022, vol. 378(6624), pp. 1092 1097.
32. Hendrycks D, Burns C., Kadavath S. et al. Measuring Mathematical Problem Solving with the MATH Dataset. arXiv preprint arXiv:2103.03874, 2021.
33. Rajkumar N., Li R., Bahdanau D. Evaluating the Text to SQL Capabilities of Large Language Models. arXiv preprint arXiv:2204.00498, 2022.
34. Olausson T.X., Inala J.P., Wang C. et al. Is Self Repair a Silver Bullet for Code Generation? arXiv preprint arXiv:2306.09896, 2023.
35. Zan D., Chen B., Zhang F. et al. Large Language Models Meet NL2Code: A Survey. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 2023, 1, pp. 7443 7464.
36. Ouyang L., Wu J., Liang X. et al. Tranining language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35, 2022, pp. 27730 27744.
37. Le H., Wang Y., Gotmare A.D. et al. CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning. Advances in Neural Information Processing Systems 35, 2022, pp. 21314 21328.
38. Shojaee P., Jain A., Tipirneni S., Reddy C.K. Execution based Code Generation using Deep Reinforcement Learning. arXiv preprint arXiv:2301.13816, 2023.
39. Liu J., Zhu Y., Xiao K. et al. RLTF: Reinforcement Learning from Unit Test Feedback. arXiv preprint arXiv:2307.04349, 2023.
40. Shen Bo, Zhang J., Chen T. et al. PanGu Coder2: Boosting Large Language Models for Code with Ranking Feedback. arXiv preprint arXiv:2307.14936, 2023.
41. Papineni K., Roukos S., Ward T., Zhu W. J. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002, pp. 311 318.
42. Lin C. Y. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization branches out, 2004, pp. 74 81.
43. Banerjee S., Lavie A. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgements. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, 2005, pp. 65 72.
44. Ren S., Guo D., Li S. et al. CodeBLEU: A Method for Automatic Evaluation of Code Synthesis. arXiv preprint arXiv:2009.10297, 2020.
45. Spiess C., Gros D., Pai K. S. et al. Calibration and Correctness of Language Models for Code. arXiv preprint arXiv:2402.02047, 2024.
46. Valentin T., Madadi A., Sapia G., Böhme M. Incoherence as Oracle-less Measure of Error in LLM-Based Code Generation. arXiv preprint arXiv:2507.00057, 2025.
47. Rice H.G. Classes of Recursively Enumerable Sets and Their Decision Problems. Transactions of the American Mathematical Society, 1953, 74(2), pp. 358-366.
48. Athiwaratkun B., Gouda S. K., Wang Z. Multi lingual Evaluation of Code Generation Models. arXiv preprint arXiv:2210.14868, 2022.
49. Liu J., Xia C. S., Wang Y., Zhang L. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. In: Oh A., Naumann T., Globerson A., Saenko K., Hardt M., Levine S. (eds.) Advances in Neural Information Processing Systems, Curran Associates, Inc., 2023. 36, pp. 21558 21572.
50. Austin J., Odena A., Nye M., Bosma M., Michalewski H., Dohan D., Jiang E., Cai C., Terry M., Le Q., Sutton C. Program Synthesis with Large Language Models. arXiv preprint arXiv:2108.07732, 2021.
51. Javaheripi M., Bubeck S., Abdin M., Aneja J., Bubeck S., Mendes C. C. T. et al. Phi-2: The surprising power of small language models. Microsoft Research Blog, 2023, 1(3).
52. Suriya G. et al. Textbooks are all you need. arXiv preprint arXiv:2306.11644, 2023.
53. Abdin M. et al. Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone. arXiv preprint arXiv:2404.14219, 2024.
54. Abdin M. et al. Phi-4 technical report. arXiv preprint arXiv:2412.08905, 2024.
55. Abouelenin A. et al. Phi-4 Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture of LoRAs. arXiv preprint arXiv:2503.01743, 2025.
56. Yang An et al. Qwen2 Technical Report. arXiv preprint arXiv:2407.10671, 2024.
57. Binyuan H. et al. Qwen2.5 Coder Technical Report. arXiv preprint arXiv:2409.12186, 2024.
58. Авагян Д.А. Реализация метрики TSA на базе набора задач HumanEval++. Доступно по ссылке: https://gitlab.com/Mathematician2000/tsa-analysis, дата обращения: 06.03.2026.
59. Авагян Д.А. Открытый набор задач HumanEval++ на HuggingFace Hub. Доступно по ссылке: https://huggingface.co/datasets/Mathematician/HumanEval-PlusPlus, дата обращения: 06.03.2026.
Рецензия
Для цитирования:
АВАГЯН Д.А. Оценка корректности сгенерированного нейросетями кода: вероятностный подход. Труды Института системного программирования РАН. 2026;38(2):111-128. https://doi.org/10.15514/ISPRAS-2026-38(2)-8
For citation:
AVAGIAN D.A. Correctness Evaluation of LLM-generated Code: Probabilistic Approach. Proceedings of the Institute for System Programming of the RAS (Proceedings of ISP RAS). 2026;38(2):111-128. (In Russ.) https://doi.org/10.15514/ISPRAS-2026-38(2)-8
Послать статью по эл. почте 