Language Models Application in Sentiment Attitude Extraction Task
https://doi.org/10.15514/ISPRAS-2021-33(3)-14
Abstract
Large text can convey various forms of sentiment information including the author’s position, positive or negative effects of some events, attitudes of mentioned entities towards to each other. In this paper, we experiment with BERT based language models for extracting sentiment attitudes between named entities. Given a mass media article and list of mentioned named entities, the task is to ex tract positive or negative attitudes between them. Efficiency of language model methods depends on the amount of training data. To enrich training data, we adopt distant supervision method, which provide automatic annotation of unlabeled texts using an additional lexical resource. The proposed approach is subdivided into two stages FRAME-BASED: (1) sentiment pairs list completion (PAIR-BASED), (2) document annotations using PAIR-BASED and FRAME-BASED factors. Being applied towards a large news collection, the method generates RuAttitudes2017 automatically annotated collection. We evaluate the approach on RuSentRel-1.0, consisted of mass media articles written in Russian. Adopting RuAttitudes2017 in the training process results in 10-13% quality improvement by F1-measure over supervised learning and by 25% over the top neural network based model results.
Keywords
Supporting agencies: This work was supported by a grant from the RFBR 20-07-01059.
About the Author
Nicolay Leonidovich RUSNACHENKO
https://nicolay-r.github.io/
Bauman Moscow State Technical University
Russian Federation
Bauman Moscow State Technical University
Russian Federation
PhD student of the department of «Theoretical Informatics and Computer Technologies»
References
1. N. Loukachevitch и N. Rusnachenko. Extracting sentiment attitudes from analytical texts. Proceedings of International Conference on Computational Linguistics and Intellectual Technologies Dialogue-2018 (arXiv:1808.08932):459—46 8, 2018.
2. M. Mintz, S. Bills, R. Snow и D. Jurafsky. Distant supervision for relation extraction without labeled data. в Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2-Volume 2, страницы 1003—1011. Association for Computational Linguistics, 2009.
3. R. Hoffmann, C. Zhang, X . Ling, L. Zettlemoyer и D. S. Weld. Knowledge-based weak supervision for information ex traction of overlapping relations. В Proceedings of the 49th annual meeting of the association for computational linguistics: hum an language technologies, страницы 541—550, 2011.
4. S. Vashishth, R. Joshi, S. S. Prayaga, C. Bhattacharyya и P. Talukdar. RESIDE: improving distantly-supervised neural relation extraction using side information: 1257—126 6, окт. 2018. url: http://aclweb.org/anthology/D18-1157.
5. N. Rusnachenko, N. Loukachevitch и E. Tutubalina. Distant supervision for sentiment attitude ex traction. в Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), страницы 1022—1030, Varna, Bulgaria. INCOMA Ltd., сент. 2019. doi: 10.26615/978-954-452-056-4_118. url: https://www.aclweb.org/anthology/R19-1118.
6. N. Loukachevitch и N. Rusnachenko. Sentiment frames for attitude ex traction in russian. Proceedings of International Conference on Com p utational Linguistics and Intellectual Technologies Dialogue-2020 (arXiv preprint arXiv:2006 .10973), 2020.
7. J. Devlin, M.-W. Chang, K. Lee и K. Toutanova. Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
8. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser и I. Polosukhin. Attention is all you need. в Advances in neural information processing system s, страницы 5998—6 008, 2017.
9. A. Radford, K. Narasimhan, T. Salimans и I. Sutskever. Improving language understanding by generative pre-training, 2018.
10. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell и др. Language models are few-shot learners. arXiv preprint arXiv:2005.1416 5, 2020.
11. C. Alt, M. Hubner и L. Hennig. Improving relation ex traction by pre-trained language representations. arXiv preprint arXiv:1906.03088, 2019.
12. C. Sun, L. Huang и X . Qiu. Utilizing bert for aspect-based sentiment analysis via constructing auxiliary sentence. в Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hum an Language Technologies, Volume 1 (Long and Short Papers), страницы 380—385, 2019.
13. Y. Kuratov и M. Arkhipov. Adaptation of deep bidirectional multilingual transformers for russian language. arXiv preprint arXiv:1905.07213, 2019.
14. S. R. Bowman, G. Angeli, C. Potts и C. D. Manning. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326 , 2015.
15. A. Conneau, G. Lample, R. Rinott, A. Williams, S. R. Bowman, H. Schwenk и V. Stoyanov. Xnli: evaluating cross-lingual sentence representations. arXiv preprint arXiv:1809.05053, 2018.
16. K. Clark, M.-T. Luong, Q. V. Le и C. D. Manning. Electra: pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555, 2020.
17. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer и V. Stoyanov. Roberta: a robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.116 92, 2019.
18. A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzman, E. Grave, M. Ott, L. Zettlemoyer и V. Stoyanov. Unsupervised cross-lingual representation learning at scale. arXiv p reprint arXiv:1911.02116 , 2019.
19. M. Joshi, D. Chen, Y. Liu, D. S. Weld, L. Zettlemoyer и O. Levy. Spanbert: improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:6 4—77, 2020.
20. I. Hendrickx, S. N. Kim, Z. Kozareva, P. Nakov, D. O S´ eaghdha, S. Pad´o, M. Pennacchiotti, L. Romano и S. Szpakowicz. Semeval-2010 task 8: multi-way classification of semantic relations between pairs of nominals. в Proceedings of the Workshop on Semantic Evaluations: Recent Achievements and Future Directions, страницы 94—99. Association for Computational Linguistics, 2009.
21. S. Wu и Y. He. Enriching pre-trained language model with entity information for relation classification. в Proceedings of the 28th ACM International Conference on Information and Knowledge Management, страницы 236 1—236 4, 2019.
22. D. Dowty. Thematic proto-roles and argument selection. language, 6 7(3):547—6 19, 1991.
23. R. Weischedel, M. Palmer, M. Marcus, E. Hovy, S. Pradhan, L. Ramshaw, N. Xue, A. Taylor, J. Kaufman, M. Franchini и др. Ontonotes release 5.0 ldc2013t19. Linguistic Data Consortium, Philadelphia, PA, 23, 2013.
24. N. Loukachevitch, G. Lashevich и B. Dobrov. Comparing two thesaurus representations for russian. в Proceedings of Global WordNet Conference GWC, страницы 35—44, 2018.
25. N. Rusnachenko и N. Loukachevitch. Neural network approach for extracting aggregated opinions from analytical articles. International Conference on Data Analytics and Management in Data Intensive Dom ains:16 7—179, 2018.
26. N. Rusnachenko и N. Loukachevitch. Attention-based neural networks for sentiment attitude ex traction using distant supervision. в The 10th International Conference on Web Intelligence, Mining and Semantics (WIMS 2020), June 30-July 3, 2020, Biarritz, France, 2020. doi: 10.1145/3405962.3405985.
27. N. Rusnachenko и N. Loukachevitch. Studying attention models in sentiment attitude ex traction task. в Proceedings of the 25th International Conference on Natural Language and Information Systems, 2020. doi: 10.1007/978-3-030-51310-8_15.
28. N. Rusnachenko и N. Loukachevitch. Using convolutional neural networks for sentiment attitude ex traction from analytical texts. EPiC Series in Language and Linguistics, 4:1—10, 2019.
29. S. Hochreiter и J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735—1780, 1997.
30. N. Loukachevitch и A. Levchik. Creating a general russian sentiment lexicon. в Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16 ), страницы 1171—1176 , 2016.
31. K. Clark, U. Khandelwal, O. Levy и C. D. Manning. What does bert look at? an analysis of bert’s attention. arXiv preprint arXiv:1906.04341, 2019.
Review
For citations:
RUSNACHENKO N.L. Language Models Application in Sentiment Attitude Extraction Task. Proceedings of the Institute for System Programming of the RAS (Proceedings of ISP RAS). 2021;33(3):199-222. (In Russ.) https://doi.org/10.15514/ISPRAS-2021-33(3)-14
Email this article 