SLAVA: Benchmark of Sociopolitical Landscape and Value Analysis
Andrey Sergeevich CHETVERGOV,
Rinat Sayarovich SHARAFETDINOV,
Marina Mikhailovna POLUKOSHKO,
Vadim Aksanovich AKHMETOV,
Natalia Andreevna ORUZHEYNIKOVA,
Egor Sergeevich ANICHKOV,
Irina Sergeevna ALEKSEEVSKAIA,
Sergey Vladimirovich BOLOVTSOV,
Pavel Evgenievich GOLOSOV https://doi.org/10.15514/ISPRAS-2025-37(3)-12
Abstract
Large Language Models (LLMs) are being applied across various fields due to their growing capabilities in numerous natural language processing tasks. However, the implementation of LLMs in systems where errors could have negative consequences necessitates a thorough examination of their reliability. Specifically, evaluating the factuality of LLMs helps determine how well the generated text aligns with real-world facts. Despite the existence of numerous factual benchmarks, only a small fraction of them assesses the models' knowledge in the Russian domain. Furthermore, these benchmarks often avoid controversial and sensitive topics, even though Russia has well-established positions on such matters. To overcome the problem of incompleteness of sensitive assessments, we have developed the SLAVA benchmark, comprising approximately 14,000 sensitive questions relevant to the Russian domain across various fields of knowledge. Additionally, for each question, we measured the provocation factor, which determines the respondent's sensitivity to the topic in question. The benchmark results allowed us to rank multilingual LLMs based on their responses to questions on significant topics such as history, political science, sociology and geography. We hope that our research will draw attention to this issue and stimulate the development of new factual benchmarks, which, through the evaluation of LLM quality, will contribute to the harmonization of the information space accessible to a wide range of users and the formation of ideological sovereignty.
Supporting agencies: Institute of Social Sciences of the Russian Presidential Academy of National Economy and Public Administration, Institute for System Programming of the Russian Academy of Sciences.
About the Authors
Andrey Sergeevich CHETVERGOV
The Russian Presidential Academy of National Economy and Public Administration
Russian Federation
Russian Federation
A specialist at the Laboratory of Intelligent Analytics at the Research Center for Artificial Intelligence, Institute for Social Sciences (ISS), Presidential Academy. Research interests include the development and optimization of machine learning models, deep learning, natural language processing, automation of ML processes, exploration of novel artificial intelligence algorithms, and interdisciplinary research.
Rinat Sayarovich SHARAFETDINOV
The Russian Presidential Academy of National Economy and Public Administration
Russian Federation
Russian Federation
A specialist at the Laboratory of Intelligent Analytics at the Research Center for Artificial Intelligence, Institute for Social Sciences (ISS), Presidential Academy. Research interests include the design and enhancement of machine learning models, the study of natural language processing methods, and the exploration of novel approaches in working with large language models (LLMs).
Marina Mikhailovna POLUKOSHKO
The Russian Presidential Academy of National Economy and Public Administration
Russian Federation
Russian Federation
The Head of the Laboratory of Intelligent Analytics at the Research Center for Artificial Intelligence, Institute for Social Sciences (ISS), Presidential Academy. Research interests include strategic management of data analysis and machine learning projects, the development and application of large language models (LLMs), trustworthiness studies of AI systems, and interdisciplinary research.
Vadim Aksanovich AKHMETOV
The Russian Presidential Academy of National Economy and Public Administration
Russian Federation
Russian Federation
An expert at the Laboratory of Intelligent Analytics at the Research Center for Artificial Intelligence, Institute for Social Sciences (ISS), Presidential Academy. Research interests include big data analysis, development of predictive models, time series analysis, computer vision, and interpretability of machine learning models.
Natalia Andreevna ORUZHEYNIKOVA
The Russian Presidential Academy of National Economy and Public Administration
Russian Federation
Russian Federation
An analyst at the Laboratory of Intelligent Analytics at the Research Center for Artificial Intelligence, Institute for Social Sciences (ISS), Presidential Academy. Research interests include big data processing, statistical modeling, data visualization, predictive analytics, and business process optimization.
Egor Sergeevich ANICHKOV
The Russian Presidential Academy of National Economy and Public Administration
Russian Federation
Russian Federation
A leading specialist at the Laboratory of Intelligent Analytics at the Research Center for Artificial Intelligence, Institute for Social Sciences (ISS), Presidential Academy. His research interests include natural language processing, scientific evaluation of projects related to large language models (LLMs), and the study of trustworthiness factors in intelligent systems.
Irina Sergeevna ALEKSEEVSKAIA
Institute for System Programming of the Russian Academy of Sciences
Russian Federation
Russian Federation
Programmer at the Trusted Artificial Intelligence Research Center, postgraduate student at the ISP RAS in the field of artificial intelligence and machine learning. Research interests: large language models, adversarial attacks, backdoor attacks, alignment of large language models.
Sergey Vladimirovich BOLOVTSOV
The Russian Presidential Academy of National Economy and Public Administration
Russian Federation
Russian Federation
The Director of the Research Center for Artificial Intelligence, Institute for Social Sciences (ISS), Presidential Academy. Research interests include the optimization and scaling of infrastructure for big data and machine learning workflows, data analysis and quality management, advanced natural language processing (NLP) methods, and the application of large language models (LLMs) in interdisciplinary research.
Pavel Evgenievich GOLOSOV
The Russian Presidential Academy of National Economy and Public Administration
Russian Federation
Russian Federation
The Director of the Institute for Social Sciences (ISS) at the Presidential Academy. Research interests include technological challenges and artificial intelligence, the data economy and AI implementation, an individualized approach in higher education, and the application of AI in education.
References
1. Minaee S. и др. Large Language Models: A Survey // 2024.
2. Wang C. и др. Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity 2023.
3. Hendrycks D. и др. Measuring Massive Multitask Language Understanding // International Conference on Learning Representations.
4. Huang Y. и др. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models // Advances in Neural Information Processing Systems. 2024. Т. 36.
5. Lin S., Hilton J., Evans O. TruthfulQA: Measuring How Models Mimic Human Falsehoods // Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022. С. 3214–3252.
6. Hu X. и др. Do Large Language Models Know about Facts? // 2023.
7. Fenogenova A. и др. MERA: A Comprehensive LLM Evaluation in Russian // 2024.
8. Shavrina T. и др. RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmark // Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020. С. 4717–4726.
9. Kukushkin A. rulm-sbs2, https://github.com/kuk/rulm-sbs2 // 2024.
10. Taktasheva E. и др. TAPE: Assessing Few-shot Russian Language Understanding // Findings of the Association for Computational Linguistics: EMNLP 2022., 2022. С. 2472–2497.
11. Открытый банк тестовых заданий [Электронный ресурс]. URL: https://ege.fipi.ru/bank/ (дата обращения: 07.11.2024).
12. ЕГЭ-2024, Математика профильного уровня: задания, ответы, решения [Электронный ресурс]. URL: https://math-ege.sdamgia.ru/ (дата обращения: 07.11.2024).
13. SLAVA: Benchmark of the Socio-political Landscape and Value Analysis, открытая часть набора данных [Электронный ресурс]. URL: https://huggingface.co/datasets/RANEPA-ai/SLAVA-OpenData-2800-v1 (дата обращения: 07.11.2024).
14. Ollama [Электронный ресурс]. URL: https://ollama.com (дата обращения: 07.11.2024).
Review
For citations:
CHETVERGOV A.S., SHARAFETDINOV R.S., POLUKOSHKO M.M., AKHMETOV V.A., ORUZHEYNIKOVA N.A., ANICHKOV E.S., ALEKSEEVSKAIA I.S., BOLOVTSOV S.V., GOLOSOV P.E. SLAVA: Benchmark of Sociopolitical Landscape and Value Analysis. Proceedings of the Institute for System Programming of the RAS (Proceedings of ISP RAS). 2025;37(3):171-184. (In Russ.) https://doi.org/10.15514/ISPRAS-2025-37(3)-12
Email this article 