Applying Language Models to Automatically Check Students’ Open-ended Answers

Automatic grading of short open-ended student answers simplifies teachers’ work and allows for quick and effective assessment. The goal of this study is to compare methods for classifying Russian-language short answers depending on the assessment. The authors analyzed the application of neural network language models and machine learning methods. Evaluation is based on a reference answer. The student’ answer is categorized in two classes: correct/incorrect, or three: correct/partially correct/incorrect. For the experiments, the authors collected four corpora of answers to questions from various disciplines and subject areas: a corpus of general questions on IT disciplines and higher mathematics, a corpus of questions on databases, a corpus of questions on history, and a corpus of questions on Qt development. During the experiments with these texts, 11 pre-trained language models, 2 training methods, 2 methods of splitting training and test sets, and 7 classifiers were compared to analyze various methods of vector representation and classification of Russian-language texts. An analysis of binary classification results revealed that there is no dominant model + classifier pair that consistently outperforms others across all corpora. BERT models in combination with a centroid classifier, logistic regression, or multilayer perceptron demonstrated the F-measure greater than 0.9. For ternary classification, the best combinations were rugpt3m, MiniLM-L12, and rubert-tiny2 models in combination with categorical boosting and a centroid classifier, with the F-measure of 0.58. Augmentation based on rules for recombination of real data helped to improve the F-measure to 0.96 for binary classification and to 0.91 for ternary classification. Error analysis revealed that the main difficulty is separating completely correct answers from partially correct ones. Based on the experimental results, a software system for conducting assessments among students was developed and published.

1. Azevedo R., Bouchet F., Duffy M., Harley J., Taub M., Trevors G., Cerezo R. Lessons learned and future directions of MetaTutor: Leveraging multichannel data to scaffold self-regulated learning with an intelligent tutoring system. Frontiers in Psychology, vol. 13, 2022, pp. 813632. DOI: 10.3389/fpsyg.2022.813632.

2. Tisha S. M., Oregon R. A., Baumgartner G., Alegre F., Moreno J. An automatic grading system for a high school-level computational thinking course. In Proc. of the 4th International Workshop on Software Engineering Education for the Next Generation, 2022, pp. 20-27. DOI: 10.1145/3528231.3528357.

3. Gao R., Merzdorf H.E., Anwar S., Hipwel M. C., Srinivasa A.R. Automatic assessment of text-based responses in post-secondary education: A systematic review. Computers and Education: Artificial Intelligence, vol. 6, 2024, pp. 100206. DOI: 10.1016/j.caeai.2024.100206.

4. Han M., Zhang X., Yuan X., Jiang J., Yun W., Gao C. A survey on the techniques, applications, and performance of short text semantic similarity. Concurrency and Computation: Practice and Experience, 33 (5), 2021, pp. e5971. DOI: 10.1002/cpe.5971.

5. Mendonça P.C., Quintal F., Mendonça F. Evaluating LLMs for automated scoring in formative assessments. Applied Sciences, 15(5), 2025, pp. 2787. DOI: 10.3390/app15052787.

6. Леонов А.Г., Мартынов Н.С., Мащенко К.А., Холькина А.А., Шляхов А.В. Автоматизация проверки семантической составляющей текстовых ответов обучающихся в цифровой образовательной платформе. Программные продукты и системы, том 37, вып. 3, 2024 г., стр. 440–452. DOI: 10.15827/0236-235X.142.440-452 / Leonov A.G., Martynov N.S., Mashchenko K.A., Kholkina A.A., Shlyakhov A.V. Automation of semantic analysis for textual responses of studentsfor textual responses of students in a digital educational platform. Software & Systems, 2024, vol. 37, issue 3, pp. 440-452 (in Russian). DOI: 10.15827/0236-235X.142.440-452.

7. Poulton A., Eliens S. Explaining transformer-based models for automatic short answer grading. In Proc. of the 5th International Conference on Digital Technology in Education, 2021, pp. 110–116. DOI: 10.1145/3488466.3488479.

8. Weegar R., Idestam-Almquist P. Reducing workload in short answer grading using machine learning. International Journal of Artificial Intelligence in Education, 34(2), 2024, pp. 247-273. DOI: 10.1007/s40593-022-00322-1.

9. Лагутина Н.С., Лагутина К.В. Обзор моделей автоматической оценки сходства ответа учащегося с эталонным ответом. Моделирование и анализ информационных систем, том 32, вып. 1, 2025 г., стр. 42-65. DOI: 10.18255/1818-1015-2025-1-42-65 / Lagutina N.S., Lagutina K.V. A survey of models for automatic assessment of similarity of student's answer to the reference answer. Modeling and Analysis of Information Systems, 2025, vol. 32, issue 1, pp. 42-65 (in Russian). DOI: 10.18255/1818-1015-2025-1-42-65.

10. Camus L., Filighera A. Investigating transformers for automatic short answer grading. In Proc. of the Artificial Intelligence in Education: 21st International Conference, 2020, pp. 43-48. DOI: 10.1007/978-3-030-52240-7_8.

11. Ahmed A., Joorabchi A., Hayes M. J. On Deep Learning Approaches to Automated Assessment: Strategies for Short Answer Grading, vol. 2, 2022, pp. 85-94. DOI: 10.5220/0011082100003182.

12. Shashavali D., Vishwjeet V., Kumar R., Mathur G., Nihal N., Mukherjee S., Patil S. V. Sentence similarity techniques for short vs variable length text using word embeddings. Computación y Sistemas, 23(3), 2019, pp. 999-1004. DOI: 10.13053/cys-23-3-3273.

13. Li H., Chu Y., Yang K., Copur-Gencturk,Y., Tang J. (2025) LLM-based Automated Grading with Human-in-the-Loop. arXiv preprint arXiv:2504.05239 (online). Доступно по ссылке: https://arxiv.org/abs/2504.05239, 01.10.2025.

14. Flodén J. Grading exams using large language models: A comparison between human and AI grading of exams in higher education using ChatGPT. British educational research journal, 51(1), 2025, pp. 201-224. DOI: 10.1002/berj.4069.

15. Jauhiainen J.S., Garagorry Guerra A. Generative AI in education: ChatGPT-4 in evaluating students’ written responses. Innovations in Education and Teaching International, 62(4), 2025, pp. 1377-1394. DOI: 10.1080/14703297.2024.2422337.

16. Ferreira M.R., Pereira J.C., Rodrigues L., Pereira F.D., Cabral L., Costa N., Ramalho G., Gasevic D. Automatic Short Answer Grading in the LLM Era: Does GPT-4 with Prompt Engineering beat Traditional Models? In Proc. of the 15th international learning analytics and knowledge conference, 2025, pp. 93-103. DOI: 10.1145/3706468.3706481.

17. Kortemeyer G. Performance of the pre-trained large language model GPT-4 on automated short answer grading. Discover Artificial Intelligence, 4(1), 2024, pp. 47. DOI: 10.1007/s44163-024-00147-y

18. Chang L.-H., Ginter F. Automatic Short Answer Grading for Finnish with ChatGPT. In Proc. of the AAAI Conference on Artificial Intelligence, 38(21), 2024, pp. 23173-23181. DOI: 10.1609/aaai.v38i21.30363

19. Bani Saad M., Jackowska-Strumillo L., Bieniecki, W. Hybrid ANN-Based and Text Similarity Method for Automatic Short-Answer Grading in Polish. Applied Sciences, 15(3), 2025, pp. 1605. DOI: 10.3390/app15031605.

20. Леонов А.Г., Мартынов Н.С., Мащенко К.А., Холькина А.А., Шляхов А.В. Автоматизация проверки семантической составляющей текстовых ответов обучающихся в цифровой образовательной платформе. Программные продукты и системы, том 37, вып. 3, 2024, стр. 440-452. DOI: 10.15827/0236-235X.142.440-452 / Leonov A.G., Martynov N.S., Mashchenko K.A., Kholkina A.A., Shlyakhov A.V. Automation of semantic analysis for textual responses of students in a digital educational platform. Software & Systems, 2024, vol. 37, Issue 3, 2024, pp. 440–452 (in Russian). DOI: 10.15827/0236-235X.142.440–452.

21. Minnegalieva, C.B., Kashapov, I.I., Morozova, O.D. Automated Grading of Students’ Short Answers Using Language Models. Automatic Documentation and Mathematical Linguistics, 58 (Suppl 3), 2024, pp. S109-S114. DOI: 10.3103/S0005105525700177.

22. Del Gobbo E., Guarino A., Cafarelli B., Grilli L., Limone P. Automatic evaluation of open-ended questions for online learning. A systematic mapping. Studies in Educational Evaluation, vol. 77, 2023, pp. 101258. DOI: 10.1016/j.stueduc.2023.101258.

23. Наборы коротких ответов с оценками. Доступно по адресу: https://gitverse.ru/Shtepser/asag-datasets, дата обращения 17.03.2026.

24. Проект «Абрис», образовательный портал факультета ИВТ. Доступно по адресу: https://abris.yarsu.ru, дата обращения: 17.03.2026. Для входа в демонстрационный режим использовать логин ‘demouser’, пароль: ‘demouser’.