Enhanced Text Classification Using DistilBERT with Low-Rank Adaptation: A Comparative Study
https://doi.org/10.15514/ISPRAS-2025-37(3)-11
Abstract
In this article, we delve into the task of sentiment analysis applied to news articles covering sanctions against Russia, with a specific focus on secondary sanctions. With geopolitical tensions influencing global affairs, understanding the sentiment conveyed in news about sanctions is crucial for policymakers, analysts, and the public alike. We explore the challenges and nuances of sentiment analysis in this context, considering the linguistic complexities, geopolitical dynamics, and data biases inherent in news reporting. Leveraging natural language processing techniques and machine learning models, including Large Language Models (LLM), 1D Convolutional Layer (Conv1D), and Feed-Forward Networks (FFN), we aim to extract sentiment insights from news articles. Our analysis provides valuable perspectives on public opinion, market reactions, and geopolitical trends. Through our work, we seek to illuminate the sentiment landscape surrounding sanctions against Russia and their broader implications.
Keywords
text classification,
sentiment analysis,
sanctions,
distilbert,
lora,
conv1d,
feed-forward networks.
Supporting agencies: This study was supported by the AI Talent Hub of ITMO University and the Russian Academy of Foreign Trade. The Russian Academy of Foreign Trade provided the dataset for the research.
About the Authors
Brice Donald ABODO ELOUNDOU
AI Talent Hub, ITMO University
Russian Federation
Russian Federation
Master student at AI Talent Hub, ITMO University. Research interests: natural language processing, machine learning, AI in medicine and geocoding.
Wang QUANYU
AI Talent Hub, ITMO University
Russian Federation
Russian Federation
Master student at AI Talent Hub, ITMO University. Research interests: deep learning, text classification, and geopolitical data analysis.
References
1. Kim, J., Weiss, J., and Wilensky, M.S.: The Political Dynamics of Sanctions: A Comparative Study, Journal of International Affairs, vol. 75, no. 2, pp. 123-145, 2021.
2. Tausczik, Y., and Pennebaker, J.: The Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods. Journal of Language and Social Psychology, vol. 29, no. 1, pp. 24-54, 2017.
3. Mearsheimer, J.: The Tragedy of Great Power Politics, W.W. Norton and Company, New York, 2001.
4. Goodfellow, I., Bengio, Y., and Courville, A.: Deep Learning. MIT Press, Cambridge, 2016.
5. Waisbord, S.: Watchdog Journalism in South America: News, Accountability, and Democracy. Columbia University Press, New York, 2000.
6. McCallum, A., and Nigam, K.: A Comparison of Event Models for Naïve Bayes Text Classification. AAAI-98 Workshop on Learning for Text Categorization, 1998.
7. LeCun, Y., Bengio, Y., and Hinton, G.: Deep Learning. Nature, vol. 521, pp. 436-444, 2015.
8. Hochreiter, S., and Schmidhuber, J.: Long Short-Term Memory. Neural Computation, vol. 9, no. 8, pp. 1735-1780, 1997.
9. Joachims, T.: Text Categorization with Support Vector Machines: Learning with Many Relevant Features. European Conference on Machine Learning, pp. 137-142, 1998.
10. Kim, Y.: Convolutional Neural Networks for Sentence Classification. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1746-1751, 2014.
11. Lai, S., Xu, L., Liu, K., and Zhao, J.: Recurrent Convolutional Neural Networks for Text Classification. Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, pp. 2267-273, 2015.
12. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I.: Attention is All You Need. Advances in Neural Information Processing Systems, vol. 30, pp. 5998- 6008, 2017.
13. Devlin, J., Chang, M-W., Lee, K., and Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of NAACL-HLT 2019, pp. 4171-4186, 2019.
14. Sanh, V., Debattista, L., Gozdz, W., Sanh, A., Chaumond, T., Lhoest, Q., Launay, J., Rush, A., and Ott, M.: DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019.
15. Hu, Z., Shen, Y., Liu, Z., and Sun, M.: Low-Rank Adaptation for Efficient Text Classification. Proceedings of ACL-IJCNLP 2021, pp. 2692-2703, 2021.
16. Strubell, E., Ganesh, A., and McCallum, A.: Energy and Policy Considerations for Deep Learning in NLP. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3645-3650, 2019.
17. Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O.: Understanding Deep Learning Requires Rethinking Generalization. Communications of the ACM, vol. 64, no. 3, pp. 107-115, 2021.
18. Doshi-Velez, F., and Kim, B.: Towards a Rigorous Science of Interpretable Machine Learning. arXiv preprint arXiv:1702.08608, 2017.
19. Hufbauer, G.C., Schott, J.J., Elliott, K.A., and Oegg, B.: Economic Sanctions Reconsidered. 3rd ed., Peterson Institute for International Economics, Washington D.C., 2007.
20. Dreger, C., Gros, K., Kooths, K., and Ulbricht, D.: The Impact of Sanctions and Oil Prices on the Russian Economy. Journal of Comparative Economics, vol. 44, no. 3, pp. 598-615, 2016.
21. Entman, R.: Framing Bias: Media in the Distribution of Power. Journal of Communication, vol. 57, no. 1, pp. 163-173, 2007.
Review
For citations:
ABODO ELOUNDOU B.D., QUANYU W. Enhanced Text Classification Using DistilBERT with Low-Rank Adaptation: A Comparative Study. Proceedings of the Institute for System Programming of the RAS (Proceedings of ISP RAS). 2025;37(3):159-170. https://doi.org/10.15514/ISPRAS-2025-37(3)-11
Email this article 