Ontology-based Neurointerface IoT Integration Approach
https://doi.org/10.15514/ISPRAS-2024-36(2)-8
Abstract
Recently, there is a surge of interest in employing neurocomputer interfaces for a control contours implementation, especially for different infrastructures of Internet of Things. However, due to a low-level nature of such devices and related software tools, neurointerface integration with a large variety of IoT devices is quite a tedious task, and the one that requires a lot of knowledge in the neuroscience and signal processing to boot. In the paper, we propose an ontology-driven solution for facing the upcoming challenges of unified integration of brain-computer interfaces into IoT ecosystems. We demonstrate an adaptable mechanism for integrating brain-computer interfaces into the Internet of Things infrastructure by introducing an intermediate layer – a smart mediator that will be responsible for communication between the environment and the neurointerface. The mediator’s software is generated automatically, and this process is driven by a managing ontology. The proposed formal model and the system's implementation are described. The approach we have developed enables researchers and engineers without strong background in brain–computer interface to automate the integration neurointerfaces with different infrastructures of Internet of Things.
Keywords
Internet of Things,
brain–computer interface,
ontology engineering,
ontology-driven solution,
smart mediator.
Supporting agencies: The authors express their deep gratitude to the Educational and Scientific Laboratory of Sociocognitive and Computer Linguistics, Faculty of Philology, Perm State University, for the provided equipment and support for the research.
About the Authors
Ivan Aleksandrovich LABUTIN
Perm State University
Russian Federation
Russian Federation
A postgraduate student of the Computer Science Department at Perm State University. His research interests include IoT and approaches for integrating neurointerfaces with IoT infrastructure, natural language processing, system programming.
Svetlana Igorevna CHUPRINA
Perm State University
Russian Federation
Russian Federation
Cand. Sci. (Phys.-Math.), Prof. of Computer Science Dept. at Perm State University, Honorary Worker of Higher Professional Education of the Russian Federation, Corresponding Member of the International Academy of Informatization. Research interests: databases and knowledge bases; expert systems as a part of the DSS, the Internet of Things, semantic web, methods and tools of ontological engineering for building intelligent systems, scientific visualization, machine learning and generative AI in NLP.
References
1. Huang S., Tognoli E. Brainware: Synergizing software systems and neural inputs. In Companion Proceedings of the 36th International Conference on Software Engineering, ser. ICSE Companion 2014, Hyderabad, India: Association for Computing Machinery, 2014, pp. 444–447, ISBN: 9781450327688. DOI: 10.1145/2591062.2591131.
2. McCullagh P., Ware M., McRoberts A., Lightbody G., Mulvenna M., McAllister G., González J. L., Medina V.C. Towards standardized user and application interfaces for the brain computer interface. Universal Access in Human-Computer Interaction. Users Diversity, C. Stephanidis, Ed., Berlin, Heidelberg: Springer Berlin Heidelberg, 2011, pp. 573–582, ISBN: 978-3642-21663-3.
3. Huggins J., Guger C., Aarnoutse E., et al. Workshops of the seventh international brain-computer interface meeting: Not getting lost in translation. Brain-Computer Interfaces, pp. 1–31, Dec. 2019. DOI: 10.1080/2326263X.2019.1697163.
4. Allison B. The I of BCIs: Next generation interfaces for brain–computer interface systems that adapt to individual users. Human-Computer Interaction. Novel Interaction Methods and Techniques, J. A. Jacko, Ed., Berlin, Heidelberg: Springer Berlin Heidelberg, 2009, pp. 558–568, ISBN: 978-3-642-02577-8.
5. Лабутин И.А., Чуприна С.И. Концепция построения онтологически управляемых нейроинтерфейсов. Интеллектуальные системы в науке и технике, 2020, с. 105-111. / Labutin I.A., Chuprina S.I. Concept of ontology-driven neurointerface development, Intellectual systems in science and technology, 2020, pp. 105–111 (in Russian).
6. Lunev D., Poletykin S., Kudryavtsev D. Brain-computer interfaces: Technology overview and modern solutions. Modern Innovations, Systems and Technologies, vol. 2, no. 3, pp. 0117–0126, Jul. 2022, ISSN: 2782-2818. DOI: 10.47813/2782-2818-2022-2-3-01170126.
7. Ryabinin K., Chuprina S., Labutin I. Ontology-driven toolset for audio-visual stimuli representation in eeg-based bci research. In Proc. of the International Conference on Computer Graphics and Vision “Graphicon”, CEUR, vol. 31, Keldysh Institute of Applied Mathematics, 2021, pp. 223–234. DOI: 10.20948/graphicon-2021-3027-223-234.
8. Gruber T.R. A translation approach to portable ontology specifications. Knowledge Acquisition, vol. 5, no. 2, pp. 199–220, 1993, ISSN: 1042-8143. DOI: https://doi.org/10.1006/knac.1993.1008.
9. Гаврилова Т., Хорошевский В. Базы знаний интеллектуальных систем: Учебник. Питер, 2000, 384 с. ISBN: 9785272000712. / Gavrilova T., Khoroshevskii V. Bazy znanii intellektual’nykh sistem: Uchebnik. Piter, 2000, 384 р. ISBN: 9785272000712 (in Russian).
10. Чуприна С.И. Адаптация технологий фабрик данных к разработке систем визуальной аналитики в области цифровой медицины. Труды 33 Международной конференции по компьютерной графике и машинному зрению «Графикон-2023», Институт проблем управления им. В.А. Трапезникова РАН, 2023, с. 405-416. / Chuprina S.I. To Adapt Data Fabric Technology to Visual Analytics Systems Development in the Field of Digital Medicine. In Proc. of 33th International Conference on Computer Graphics and Machine Vision “GrafiCon-2023”, V.A. Trapeznikov Institute of Control Sciences of RAS, 2023, pp. 405–416. DOI: 10.20948/graphicon-2023-405-416 (in Russian).
11. Chuprina S.I., Ryabinin K.V., Koznov D.V., Matkin K.A. Ontology-driven visual analytics software development, Programming and Computer Software, vol. 48, no. 3, pp. 208–214, Jun. 2022, ISSN: 1608-3261. DOI: 10.1134/S0361768822030033.
12. Чуприна С.И., Рябинин К.В., Кознов Д.В., Маткин К.А. Онтологически управляемые средства автоматизации разработки приложений визуальной аналитики, Программирование, т. 3, 2022, с. 70-77. / Chuprina S.I., Ryabinin K.V., Koznov D.V., Matkin K.A. Ontologicheski upravlyaemye sredstva avtomatizatsii razrabotki prilozhenii vizual’noi analitiki, Programmirovanie, no. 3, 2022, pp. 70–77 (in Russian).
13. Chuprina S.I. Using data fabric architecture to create personalized visual analytics systems in the field of digital medicine, Scientific Visualization, vol. 15(5), 2023, pp. 50–63. DOI: 10.26583/sv.15.5.05.
14. Ryabinin K., Chuprina S., Belousov K. Ontology-driven automation of IOT-based human-machine interfaces development. In Computational Science – ICCS 2019, J. M. F. Rodrigues, P. J. S. Cardoso, J. Monteiro, et al., Eds., Cham: Springer International Publishing, 2019, pp. 110–124, ISBN: 978-3-030-22750-0.
15. Mendez S.J.R., Zao J.K. BCI ontology: A context-based sense and actuation model for brain-computer interactions. In Proc. of the 9th International Semantic Sensor Networks Workshop co-located with 17th International Semantic Web Conference, 2018, pp. 32-47.
16. Ryabinin K., Chuprina S. High-level toolset for comprehensive visual data analysis and model validation, Procedia Computer Science, vol. 108, pp. 2090– 2099, Dec. 2017. DOI: 10.1016/j.procs.2017.05.050.
17. Ryabinin K., Chuprina S., Kolesnik M. Calibration and monitoring of IOT devices by means of embedded scientific visualization. In Proc/ of International Conference on Conceptual Structures, 2018, pp. 655-668. DOI: 10.1007/978-3-319-93701-4_52.
18. Izumigawa C., Taylor B., Sato J. Automated ontology generation. In HCI International 2023 Posters, C. Stephanidis, M. Antona, S. Ntoa, and G. Salvendy, Eds., Cham: Springer Nature Switzerland, 2023, pp. 433–438, ISBN: 978-3-031-36004-6.
19. Elnagar S., Yoon V.Y., Thomas M.A. An automatic ontology generation framework with an organizational perspective. CoRR, vol. abs/2201.05910, 2022. arXiv: 2201.05910.
20. Sassi N., Jaziri W., Gargouri F. How to evolve ontology and maintain its coherence - a corrective operations-based approach. In Proc. of the First International Conference on Knowledge Engineering and Ontology Development, 2009, pp. 384–387.
21. Jaziri W., Sassi N., Gargouri F. Approach and tool to evolve ontology and maintain its coherence. International Journal of Metadata Semantics and Ontologies, vol. 5, pp. 151–166, May 2010. DOI: 10.1504/IJMSO. 2010.033284.
22. Рябинин К.В. Методы и средства разработки адаптивных мультиплатформенных систем визуализации научных экспериментов. Диссертация на соискание ученой степени кандидата физико-математических наук. ИПМ им. М.В. Келдыша, Москва, 2015, 208 c./ Ryabinin K. Metody i sredstva razrabotki adaptivnykh mul’tiplatformennykh sistem vizualizatsii nauchnykh eksperimentov. PhD Thesis, IPM im. M.V. Keldysha, Moskva, 2015, 208 p. (in Russian).
23. Seydoux N., Drira K., Hernandez N., Monteil T. IOT-O, a core-domain IOT ontology to represent connected devices networks. In Knowledge Engineering and Knowledge Management, E. Blomqvist, P. Ciancarini, F. Poggi, and F. Vitali, Eds., Cham: Springer International Publishing, 2016, pp. 561–576, ISBN: 9783-319-49004-5.
24. Чуприна С.И., Зиненко Д.В. Онтолис: Адаптируемый визуальный редактор онтологий. Вестник Пермского Университета. Серия: Математика. Механика. Информатика, том 3 (22), 2013, с. 106-110. / Chuprina S.I., Zinenko D.V. Adaptable Visual Ontological Editor ONTOLIS. Vestnik Permskogo universiteta. Seriya: Matematika. Mekhanika. Informatika, vol. 3 (22), 2013, pp. 106–110 (in Russian).
25. Chuprina S., Nasraoui O. Using ontology-based adaptable scientific visualization and cognitive graphics tools to transform traditional information systems into intelligent systems. Scientific Visualization, vol. 8 (1), pp. 23–44, Jan. 2016.
26. Лабутин И.А. Система генерации программного обеспечения для устройств интернета вещей на базе онтологического описания инфраструктуры. 17.01.2023, РОСПАТЕНТ №2023612016 / Labutin I.A. Sistema generatsii programmnogo obespecheniya dlya ustroistv interneta veshchei na baze ontologicheskogo opisaniya infrastruktury. 17.06.2023. ROSPATENT №2023612016 (in Russian).
Review
For citations:
LABUTIN I.A., CHUPRINA S.I. Ontology-based Neurointerface IoT Integration Approach. Proceedings of the Institute for System Programming of the RAS (Proceedings of ISP RAS). 2024;36(2):91-108. https://doi.org/10.15514/ISPRAS-2024-36(2)-8
Email this article 