Experimental Study of Instruction-Based Models for Extracting Domain-Specific Entities from Student Reports

This work investigated the task of extracting domain-specific entities from student reports in the field of information technology. Domain-specific entities (DSE) represent key terms, skills, and named entities that reflect the thematic specifics of the text. The solutions evaluated included the keyword extraction tool rutermextract, a fine-tuned mBART language model, and instruction-tuned large language models (YandexGPT, Saiga, Tlite). The study found that fine-tuning mBART is effective given a sufficient volume of data. Instruction-based models outperformed rutermextract and show promise for low-data scenarios, with the Saiga model being particularly effective at identifying the core set of entities. The strategy of highlighting domain-specific entities within the text was found to be more accurate than extracting them as a simple list. However, the task requires further research: the high rate of erroneous extraction of domain-specific entities (67-89%), manifested as a complete lack of overlap with the gold-standard entities, indicates the models' difficulty in separating the core entity from its context. The main limitations of the study are the small corpus size (2,933 texts) and the use of simple instructions. Promising research directions include developing more detailed instructions and evaluating the approaches in other domains and text types.

1. Кан А. В., Козловская Я. Д., Токолова А. А. Извлечение научно-технических фактов из отраслевых документов на основе методов семантико-синтаксического и концептуального анализа. Моделирование и анализ данных, 2024, т. 14, № 1, с. 27-40. / Kan A.V. et al. Izvlechenie nauchno-tekhnicheskikh faktov iz otraslevykh dokumentov na osnove metodov semantiko-sintaksicheskogo i kontseptual'nogo analiza [Extraction of scientific and technical facts from industry documents based on semantic-syntactic and conceptual analysis methods]. Modelirovanie i analiz dannykh [Modeling and Data Analysis], 2024, vol. 14, no. 1, pp. 27-40 (in Russian). DOI: 10.17759/mda.2024140102.

2. Chumwatana T., Hpone A. K. K. Bridging the IT skill gap with industry demands: An AI-driven text mining approach to job market trends using large language model. Journal of Theoretical and Applied Information Technology, 2025, vol. 103, no. 6, pp. 2270-2282. DOI: 10.5281/zenodo.17175903.

3. Мельникова А. В. и др. Разработка алгоритма формирования команд ИТ-проектов на основе данных цифрового следа студентов. Труды Института системного программирования РАН, 2024, т. 36, № 3, с. 213-224. / Melnikova A.V. et al. Razrabotka algoritma formirovaniya komand IT-proektov na osnove dannykh tsifrovogo sleda studentov [Development of an algorithm of the formation of IT project teams based on data from the digital footprint of students]. Trudy Instituta sistemnogo programmirovaniya RAN [Proceedings of the Institute for System Programming of the RAS], 2024, vol. 36, no. 3, pp. 213-224 (in Russian). DOI: 10.15514/ISPRAS-2024-36(3)-15.

4. Мельникова А. В., Воробьева М. С., Глазкова А. В. Сравнение предварительно обученных моделей для извлечения предметно-ориентированных сущностей из студенческих отчетных документов // Моделирование и анализ информационных систем, 2025, т. 32, № 1, с. 66-79. / Melnikova A.V., Vorobeva M.S., Glazkova A.V. Sravnenie predvaritelno obuchennykh modelei dlya izvlecheniya predmetno-orientirovannykh sushchnostei iz studencheskikh otchetnykh dokumentov [Comparison of pre-trained models for domain-specific entity extraction from student report documents]. Modelirovanie i analiz informatsionnykh sistem [Modeling and Analysis of Information Systems], 2025, vol. 32, no. 1, pp. 66-79 (in Russian). DOI: 10.18255/1818-1015-2025-1-66-79.

5. Rutermextract. Available at: https://github.com/igor-shevchenko/rutermextract (accessed: 29.09.2025).

6. Бутенко Ю. И., Николаева Н. С., Маргарян Т. Д. Структурные модели терминологических словосочетаний для разметки корпуса научно-технических текстов. Вестник Новосибирского государственного университета. Серия: Лингвистика и межкультурная коммуникация, 2021, т. 19, № 3, с. 45-56. / Butenko Yu.I., Nikolaeva N.S., Margaryan T.D. Strukturnye modeli terminologicheskikh slovosochetanii dlya razmetki korpusa nauchno-tekhnicheskikh tekstov [Structural models of terminological word combinations for marking up a corpus of scientific and technical texts]. Vestnik Novosibirskogo gosudarstvennogo universiteta. Seriya: Lingvistika i mezhkulturnaya kommunikatsiya [Bulletin of Novosibirsk State University. Series: Linguistics and Intercultural Communication], 2021, vol. 19, no. 3, pp. 45-56 (in Russian). DOI:10/25205/1818-7935-2021-19-3-45-56.

7. Бутенко Ю. И. Извлечение номенклатурных наименований из англо-и русскоязычных научно-технических текстов // Искусственный интеллект и принятие решений, 2024, № 3, с. 113-121. / Butenko Yu.I. Izvlechenie nomenklaturnykh nazvanii iz anglo- i russkoyazychnykh nauchno-tekhnicheskikh tekstov [Nomenclature names extraction from English and Russian-language scientific and technical texts]. Iskusstvennyi intellect i prinyatie reshenii [Artificial Intelligence and Decision Making], 2024, no. 3, pp. 113-121 (in Russian). DOI: 10.14357/20718594240309.

8. Шелманов А. О. и др. Открытое извлечение информации из текстов. Часть II. Извлечения семантических отношений с помощью машинного обучения без учителя // Искусственный интеллект и принятие решений, 2019, № 2, c. 39-49. / Shelmanov A.O. et al. Otkrytoe izvlechenie informatsii iz tekstov. Chast II. Izvlecheniya semanticheskikh otnoshenii s pomoshchyu mashinnogo obucheniya bez uchitelya [Open information extraction from texts. Part II. Extraction of semantic relations using unsupervised machine learning] // Iskusstvennyi intellect i prinyatie reshenii [Artificial Intelligence and Decision Making], 2019, no. 2, pp. 39-49 (in Russian). DOI: 10.14357/20718594190204.

9. Бручес Е. П., Батура Т. В. Метод автоматического извлечения терминов из научных статей на основе слабо контролируемого обучения. Вестник Новосибирского государственного университета. Серия: Информационные технологии, 2021, т. 19, № 2, с. 5-16. / Bruches E.P., Batura T.V. Metod avtomaticheskogo izvlecheniya terminov iz nauchnykh statei na osnove slabo kontroliruemogo obucheniya [Method for automatic term extraction from scientific articles based on weak supervision] // Vestnik Novosibirskogo gosudarstvennogo universiteta. Seriya: Informatsionnye tekhnologii [Bulletin of Novosibirsk State University. Series: Information Technologies], 2021, vol. 19. no. 2, pp. 5-16 (in Russian). DOI: 10/25205/1818-7900-2021-19-2-5-16.

10. Devlin J. et al. BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), 2019, pp. 4171-4186. DOI: 10.18653/v1/N19-1423.

11. Дементьева Я. Ю., Бручес Е. П., Батура Т. В. Извлечение терминов из текстов научных статей // Программные продукты и системы, 2022, т. 35, № 4, с. 689-697. / Dementieva Ya.Yu., Bruches E.P., Batura T.V. Izvlechenie terminov iz tekstov nauchnykh statei [Terms extraction from texts of scientific papers]. Programmye produkty i sistemy [Software & Systems], 2022, vol. 35, no. 4, pp. 689-697 (in Russian). DOI: 10.15827/0236-235X.140.689-697.

12. Большакова Е. И., Семак В. В. Методы и средства извлечения терминов из текстов для терминологических задач. Программные продукты и системы, 2025, т. 38, № 1, с. 5-16. / Bolshakova E.I., Semak V.V. Metody i sredstva izvlecheniya terminov iz tekstov dlya terminologicheskikh zadach [Methods and means of term extraction from texts for terminological tasks]. Programmye produkty i sistemy [Software & Systems], 2025, vol. 38, no. 1, pp. 5-16 (in Russian). DOI: 10.15827/0236-235X.149.005-016.

13. Mamontova A., Ischenko R., Vorontsov K. RuTermEval-2024: Cross-domain Automatic Term Extraction and Classification in Russian scientific texts. Proceedings of the International Conference «Dialogue», 2025, vol. 2025, pp. 245-256. DOI: 10.28995/2075-7182-2025-23-245-256.

14. Rozhkov I., Loukachevitch N. Methods for Recognizing Nested Terms. Proceedings of the International Conference «Dialogue», 2025, vol. 2025, pp. 299-311. DOI: 10.48550/arXiv.2504.16007.

15. Korytov P. V. et al. Analysis of approaches for identifying key skills in vacancies. 2024 XXVII International Conference on Soft Computing and Measurements (SCM). IEEE, 2024, pp. 242-245. DOI: 10.1109/SCM62608.2024.10554269.

16. Николаев И. Е. Метод извлечения знаний и навыков/компетенций из текстов требований вакансий. Онтология проектирования, 2023, т. 13, № 2 (48), с. 282-293. / Nikolaev I.E. Metod izvlecheniya znanii i navykov/kompetentsii iz tekstov trebovanii vakansii [Knowledge and skills extraction from the job requirements texts]. Ontologiya proektirovaniya [Ontology of Designing], 2023, vol. 13, No. 2 (48), pp. 282-293 (in Russian). DOI: 10.18287/2223-9537-2023-13-2-282-293.

17. Matkin N. et al. Comparative Analysis of Encoder-Based NER and Large Language Models for Skill Extraction from Russian Job Vacancies // Analysis of Images, Social Networks and Texts: 12th International Conference, AIST 2024, Bishkek, Kyrgyzstan, October 17–19, 2024, Revised Selected Papers. Springer Nature, 2025, vol. 2364, pp. 45-51. DOI: 10.48550/arXiv.2407.19816.

18. Khokhlova M., Koryshev M. Keyness Analysis and Its Representation in Russian Academic Papers on Computational Linguistics: Evaluation of Algorithms. RASLAN, 2022, pp. 25-33.

19. Митрофанова О. А., Гаврилик Д. А. Эксперименты по автоматическому выделению ключевых выражений в стилистически разнородных корпусах русскоязычных текстов. Terra Linguistica, 2022, т. 13, № 4, с. 22-40. / Mitrofanova O.A., Gavrilik D.A. Eksperimenty po avtomaticheskomu vydeleniyu klyuchevykh vyrazhenii v stilisticheski raznorodnykh korpusakh russkoyazychnykh tekstov [Experiments on automatic keyphrase extraction in stylistically heterogeneous corpus of Russian texts]. Terra Linguistica, 2022, vol. 13, no. 4, pp. 22-40 (in Russian). DOI: 10.18721/JHSS.13402.

20. Song M., Feng Y., Jing L. A survey on recent advances in keyphrase extraction from pre-trained language models. Findings of the association for computational linguistics: EACL 2023, 2023, pp. 2153-2164. DOI: 10.18653/v1/2023.findings-eacl.161.

21. Glazkova A. V. et al. Keyword generation for Russian-language scientific texts using the mT5 model. Automatic Control and Computer Sciences, 2024, vol. 58, no. 7, pp. 995-1002. DOI: 10.3103/S014641162470041X.

22. Гусева Д. Д., Митрофанова О. А. Ключевые выражения в русскоязычных научно-популярных текстах: сравнение восприятия устной и письменной речи с результатами автоматического анализа. Terra Linguistica, 2024, т. 15, № 1, с. 20-35. / Guseva D.D., Mitrofanova O.A. Klyuchevyye vyrazheniya v russkoyazychnykh nauchno-populyarnykh tekstakh: sravneniye vospriyatiya ustnoy i pis'mennoy rechi s rezultatami avtomaticheskogo analiza [Keyphrases in Russian-language popular science texts: comparison of oral and written speech perception with the results of automatic analysis]. Terra Linguistica, 2024, vol. 15, no. 1, pp. 20-35 (in Russian). DOI: 10.18721/JHSS.15102.

23. Breton J. et al. Leveraging LLMs for legal terms extraction with limited annotated data. Artificial Intelligence and Law. 2025, pp. 1-27. DOI: 10.1007/s10506-025-09448-8.

24. Nguyen K. C. et al. Rethinking Skill Extraction in the Job Market Domain using Large Language Models. 1st Workshop on Natural Language Processing for Human Resources, NLP4HR 2024. Association for Computational Linguistics, ACL Anthology, 2024, pp. 27-42. Available at: https://arxiv.org/pdf/2402.03832, accessed 29.09.2025.

25. Yandex YandexGPT-5-Lite-8B-pretrain. Available at: https://huggingface.co/yandex/YandexGPT-5-Lite-8B-pretrain, accessed: 29.09.2025.

26. saiga_llama3_8b Available at: https://huggingface.co/IlyaGusev/saiga_llama3_8b, accessed: 29.09.2025.

27. Grattafiori A. et al. The Llama 3 herd of models, 2024. Available at: https://arxiv.org/pdf/2407.21783, accessed 29.09.2025.

28. Tang Y. et al. Multilingual translation from denoising pre-training. Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, 2021, pp. 3450-3466. DOI: 10.18653/v1/2021.findings-acl.304.

29. Lewis M. et al. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 7871-7880. DOI: 10.18653/v1/2020.acl-main.703.