Исправление грамматических ошибок на уровне подслов: универсальный подход

В данном исследовании мы предлагаем полностью автоматическую методологию генерации данных, построения словаря правил исправления и обучения модели разметки последовательностей, специально ориентированную на исправление грамматических ошибок. Наш подход работает на уровне подслов SentencePiece, используя базовые преобразования – сохранение, добавление, замену и удаление, которые универсально применимы во всех языках, тем самым устраняя необходимость в грамматически-специфичных операциях. Используя алгоритм Левенштейна для генерации истинных исправлений и редакционных предписаний, мы получили полностью неконтролируемый и независимый от языка процесс генерации наборов данных. Мы применили наш метод к модели разметки последовательностей GECToR и достигли сопоставимых результатов качества для английского языка с оценками F_0.5 62,4 на CoNLL-2014 (тестовый набор) и 61,9 на BEA-2019 (тестовый набор), без ручного конструирования правил и аннотации типов ошибок. Это показывает, что универсальные правила на уровне подслов могут стать альтернативой грамматическим операциям, при этом требуя только параллельные тексты – с ошибками и без ошибок.

1. Rozovskaya A., Roth D. Grammatical Error Correction: Machine Translation and Classifiers. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, 2016, pp. 2205-2215.

2. Yuan Z., Stahlberg F., Rei M., Byrne B., Yannakoudakis H. Neural and FST-based approaches to grammatical error correction. Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, Florence, Italy, 2019, pp. 228-239.

3. Bryant C., Ng H. How Far are We from Fully Automatic High Quality Grammatical Error Correction? Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, Beijing, China, 2015, pp. 697-707.

4. Patel H., Rajput D., Gadekallu T., Iwendi C., Bashir A., Jo H. A review on classification of imbalanced data for wireless sensor networks. International Journal of Distributed Sensor Networks, vol. 16, 2020.

5. Flickinger D., Yu J. Toward More Precision in Correction of Grammatical Errors. Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Shared Task, Sofia, Bulgaria, 2013, pp. 68-73.

6. Yuan X., Pham D., Davidson S., Yu Z. ErAConD: Error Annotated Conversational Dialog Dataset for Grammatical Error Correction. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Seattle, United States, 2022, pp. 76-84.

7. Lee J., Seneff S. An analysis of grammatical errors in non-native speech in english. 2008 IEEE Spoken Language Technology Workshop, 2008, pp. 89-92.

8. Ng H., Wu S., Briscoe T., Hadiwinoto C., Susanto R. Bryant C. Encode, The CoNLL-2014 Shared Task on Grammatical Error Correction. Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task, Baltimore, Maryland, 2014, pp. 1-14.

9. Omelianchuk K., Atrasevych V., Chernodub A., Skurzhanskyi O. GECToR – Grammatical Error Correction: Tag, Not Rewrite. Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, Seattle, WA, USA, 2020, pp. 163-170.

10. Khabutdinov I., Chashchin A., Grabovoy A., Kildyakov A., Chekhovich Yu. RuGECToR: Rule-Based Neural Network Model for Russian Language Grammatical Error Correction. Program. Comput. Softw., vol. 50, issue 4, 2024, pp. 315-321.

11. Malmi E., Krause S., Rothe S., Mirylenka D., Severyn A. Encode, Tag, Realize: High-Precision Text Editing. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, 2019, pp. 5054-5065.

12. Rothe S., Mallinson J., Malmi E., Krause E., Severyn A. A Simple Recipe for Multilingual Grammatical Error Correction. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, 2021, pp. 702 707.

13. Grundkiewicz R., Junczys-Dowmunt M., Heafield K. Neural Grammatical Error Correction Systems with Unsupervised Pre-training on Synthetic Data. Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, Florence, Italy, 2019, pp. 252-263.

14. Rozovskaya A., Roth D. Grammar Error Correction in Morphologically Rich Languages: The Case of Russian. Transactions of the Association for Computational Linguistics, vol. 7, 2019, pp. 1--17.

15. Kudo T., Richardson J. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Brussels, Belgium, 2018, pp. 66-71.

16. Limisiewicz T., Balhar J., Marecek D. Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languages. Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, 2023, pp. 5661-5681.

17. Asvarov A., Grabovoy A. The Impact of Multilinguality and Tokenization on Statistical Machine Translation. 35th Conference of Open Innovations Association (FRUCT), Tampere, Finland, 2024, pp. 149-157.

18. Liu Y., Ott M., et. al. RoBERTa: A Robustly Optimized BERT Pretraining Approach, 2019. Cite arxiv:1907.11692.

19. Devlin J., Chang M., Lee K., Toutanova K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, Minnesota, 2019, pp. 4171-4186.

20. Yang Z., Dai Z., Yang Y., Carbonell J., Salakhutdinov R., Le Q. XLNet: generalized autoregressive pretraining for language understanding. Proceedings of the 33rd International Conference on Neural Information Processing Systems, Red Hook, NY, USA, 2019.

21. Dahlmeier D., Ng H. Better Evaluation for Grammatical Error Correction. Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Montreal, Canada, 2012, pp. 568-572.

22. Awasthi A., Sarawagi S., et. al. Parallel Iterative Edit Models for Local Sequence Transduction. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, 2019, pp. 4259-4269.

23. Mizumoto T., Komachi M., Nagata M., Matsumoto J. Mining revision log of language learning SNS for automated Japanese error correction of second language learners. Proc. of 5th International Joint Conference on Natural Language Processing, 2011, pp. 147-155.

24. Dahlmeier D., Ng H., Wu S. Building a Large Annotated Corpus of Learner English: The NUS Corpus of Learner English. Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications, Atlanta, Georgia, 2013, pp. 22-31.

25. Yannakoudakis H., Briscoe T., Medlock B. A New Dataset and Method for Automatically Grading ESOL Texts. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Portland, Oregon, USA, 2011, pp. 180-189.

26. Bryant C., Felice M., et. al. The BEA-2019 Shared Task on Grammatical Error Correction. Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, Florence, Italy, 2019, pp. 52-75.

27. Bryant C., Felice M., Briscoe T. Automatic Annotation and Evaluation of Error Types for Grammatical Error Correction. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics,Vancouver, Canada, 2017, pp. 793-805.