От простого взаимодействия к персонализированному обучению: исследование взаимодействий пользователя с объектом в интеллектуальном окружении

Цель этой работы состоит в развитии степени персонализации интеллектуальных учебных сред путем анализа данных взаимодействия пользователя с объектом, чтобы определить его поведение при решении задачи и в перерывах в работе (On-Task/Off-Task). Это достигается путем мониторинга и анализа взаимодействия пользователей при выполнении учебных заданий с гибридной системой, адаптированной для интеллектуальной университетской среды. Предложенная конфигурация системы, дополненная инструментарием добычи данных Orange и классификаторами на основе моделей нейронной сети, случайного леса, наивного классификатора Байеса и дерева классификации было проведено обучение взаимодействиям с объектами и тестирование 13 студентов (11 для обучения и два для тестирования), что позволило выявить представительные последовательности действий. Несмотря на небольшое количество данных, удалось понять, что наилучшие результаты показали две модели –нейронная сеть и наивный классификатор Байеса. Хотя для адекватного выполнения классификации необходим более значительный объем данных, проведенный опыт позволил лучше понять процесс. Впоследствии его можно будет полностью включить в интеллектуальную образовательную систему, что позволит внести вклад в создание персонализированных сред.

1. G. Virone et al., Behavioral Patterns of Older Adults in Assisted Living, in IEEE Transactions on Information Technology in Biomedicine, vol. 12, no. 3, pp. 387-398, May 2008, doi: 10.1109/TITB.2007.904157.

2. Pantic Maja, et al. Human computing and machine understanding of human behavior: A survey. Proceedings of the 8th international conference on Multimodal Interfaces. 2006.

3. Favela Jesus. Activity, Behavior, and Context: The ABC of Pervasive Healthcare Research. Intelligent Environments (Workshops). 2012.

4. Nef Tobias et al. Evaluation of three state-of-the-art classifiers for recognition of activities of daily living from smart home ambient data. Sensors 15.5 (2012): 11725-11740.

5. Vinh La The, et al. Semi-Markov conditional random fields for accelerometer-based activity recognition. Applied Intelligence 35 (2011): 226-241.

6. Moore Dennis W. et al. Increasing on-task behavior in students in a regular classroom: Effectiveness of a self-management procedure using a tactile prompt. Journal of Behavioral Education 22 (2013): 302-311.

7. Godwin Karrie E. et al. Off-task behavior in elementary school children. Learning and Instruction 44 (2016): 128-143.

8. Doddannara Lakshmi S. et al. Exploring the relationships between design, students' affective states, and disengaged behaviors within an ITS. Artificial Intelligence in Education: 16th International Conference, AIED 2013, Memphis, TN, USA, July 9-13, 2013. Proceedings 16. Springer Berlin Heidelberg, 2013.

9. Wang Li‐Chun, Ming‐Puu Chen. The effects of game strategy and preference‐matching on flow experience and programming performance in game‐based learning. Innovations in Education and Teaching International 47.1 (2010): 39-52.

10. Schaufeli Wilmar B., Arnold Bakker. UWES Utrecht work engagement scale preliminary manual. Occupational Health Psychology Unit (2003).

11. Bumbacco C., Scharfe E. (2023). Why attachment matters: first-year post-secondary students' experience of burnout, dIsengagement, And drop-out. Journal of College Student Retention: Research, Theory & Practice, 24(4), 988-1001.

12. Zhang J., Mohd Yunos Z, Haron H. Interactivity Recognition Graph Neural Network (IR-GNN) Model for Improving Human–Object Interaction Detection. Electronics. 2023; 12(2):470. https://doi.org/10.3390/electronics12020470

13. Qing Ye, Xiuju Xu, Human-object interaction detection based on graph model, Proc. SPIE 12610, Third International Conference on Artificial Intelligence and Computer Engineering (ICAICE 2022), 126100A (28 April 2023); https://doi.org/10.1117/12.2671248

14. Mazzamuto Michele, et al. A wearable device application for human-object interactions detection. International Conference on Computer Vision Theory and Applications (VISAPP). 2023.

15. Luo Tianlun, et al. From detection to understanding: A survey on representation learning for human-object interaction. Neurocomputing (2023): 126243.

16. Kim, Sanghyun, Deunsol Jung, Minsu Cho. Relational Context Learning for Human-Object Interaction Detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

17. Ye Qing, et al. Human object interaction detection based on feature optimization and key human-object enhancement. Journal of Visual Communication and Image Representation 93 (2023): 103824.

18. Zidianakis Emmanouil, et al. Building a sensory infrastructure to support interaction and monitoring in ambient intelligence environments. Distributed, Ambient, and Pervasive Interactions: Second International Conference, DAPI 2014, Held as Part of HCI Interational 2014, Heraklion, Crete, Greece, June 22-27, 2014. Proceedings 2. Springer International Publishing, 2014.

19. D'Mello Sidney K. et al. Multimodal capture of teacher-student interactions for automated dialogic analysis in live classrooms. Proceedings of the 2015 ACM on international conference on multimodal interaction. 2015.

20. Belapurkar Neha, Sagar Shelke, Baris Aksanli. The case for ambient sensing for human activity detection. Proceedings of the 8th International Conference on the Internet of Things. 2018.

21. Jayarajah Kasthuri et al. Need accurate user behaviour? pay attention to groups!. Proceedings of the 2015 ACM international joint conference on pervasive and ubiquitous computing. 2015.

22. Baker Ryan Shaun et al. Off-task behavior in the cognitive tutor classroom: When students "Game the system". Proceedings of the SIGCHI conference on Human factors in computing systems. 2004.

23. Baker Ryan. Modeling and understanding students' off-task behavior in intelligent tutoring systems. Proceedings of the SIGCHI conference on Human factors in computing systems. 2007.

24. Leonidis Asterios et al. An intelligent task assignment and personalization system for students' online collaboration. Universal Access in Human-Computer Interaction. Applications and Services: 6th International Conference, UAHCI 2011, Held as Part of HCI International 2011, Orlando, FL, USA, July 9-14, 2011, Proceedings, Part IV 6. Springer Berlin Heidelberg, 2011.

25. Sanish Rai, Xiaolin Hu. Behavior pattern detection for data assimilation in agent-based simulation of smart environments. 2013 IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT). Vol. 2. IEEE, 2013.

26. Wang Qunbo, Wenjun Wu, Yuxing Qi. A Learning Analytic Model for Smart Classroom. Web and Big Data: APWeb-WAIM 2018 International Workshops: MWDA, BAH, KGMA, DMMOOC, DS, Macau, China, July 23–25, 2018, Revised Selected Papers 2. Springer International Publishing, 2018.

27. Korozi Maria et al. LECTOR: towards reengaging students in the educational process inside smart classrooms. Intelligent Human Computer Interaction: 9th International Conference, IHCI 2017, Evry, France, December 11-13, 2017, Proceedings 9. Springer International Publishing, 2017.

28. White Ryen W., Ahmed Hassan Awadallah, Robert Sim. Task completion detection: A study in the context of intelligent systems. Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2019.

29. Montebello Matthew. Assisting education through real-time learner analytics. 2018 IEEE Frontiers in Education Conference (FIE). IEEE, 2018.

30. Matsui Kanae, Tatsuhiko Kasai, Keiya Sakai. Challenges for data collecting of teacher and student'behavior in different types of class using video and wearable device. 2019 Joint 8th International Conference on Informatics, Electronics & Vision (ICIEV) and 2019 3rd International Conference on Imaging, Vision & Pattern Recognition (icIVPR). IEEE, 2019.

31. Prabono Aria Ghora, Seok-Lyong Lee, Bernardo Nugroho Yahya. Context-based similarity measure on human behavior pattern analysis. Soft Computing 23 (2019): 5455-5467.

32. Biswas Gautam, Brian Sulcer. Visual exploratory data analysis methods to characterize student progress in intelligent learning environments. 2010 International Conference on Technology for Education. IEEE, 2010.

33. G. Mathioudakis et al. Amiria: Real-time teacher assistance tool for an ambient intelligence classroom. Proceedings of the Fifth International Conference on Mobile, Hybrid, and Online Learning (eLmL 2013). 2013.

34. Barbara Bruno et al. Analysis of human behavior recognition algorithms based on acceleration data. 2013 IEEE International Conference on Robotics and Automation. IEEE, 2013.

35. Valérian Guivarch et al. Hybrid system to analyze user's behaviour. 2016 IEEE Symposium Series on Computational Intelligence (SSCI). IEEE, 2016.

36. Louis Atallah, Guang-Zhong Yang. The use of pervasive sensing for behaviour profiling – a survey. Pervasive and mobile computing 5.5 (2009): 447-464.

37. Zacharoula Papamitsiou, Eirini Karapistoli, Anastasios A. Economides. Applying classification techniques on temporal trace data for shaping student behavior models. Proceedings of the sixth international conference on learning analytics & knowledge. 2016.

38. Valeria Soto-Mendoza, et al. Detecting abnormal behaviours of institutionalized older adults through a hybrid-inference approach. Pervasive and Mobile Computing 40 (2017): 708-723.

39. Andreas D. Lattner, et al. Sequential pattern mining for situation and behavior prediction in simulated robotic soccer. RoboCup 2005: Robot Soccer World Cup IX 9. Springer Berlin Heidelberg, 2006.

40. Magnus S. Magnusson. Discovering hidden time patterns in behavior: T-patterns and their detection. Behavior research methods, instruments, & computers 32.1 (2000): 93-110.

41. Maurizio Casarrubea et al. T-pattern analysis for the study of temporal structure of animal and human behavior: a comprehensive review. Journal of neuroscience methods 239 (2015): 34-46.

42. Carlos Santoyo et al. Observational analysis of the organization of on-task behavior in the classroom using complementary data analyses. Anales de Psicología, 2017, vol. 33, num. 3, p. 497-514 (2017).

43. Ramakrishnan Srikant, Rakesh Agrawal. Mining sequential patterns: Generalizations and performance improvements. Advances in Database Technology—EDBT'96: 5th International Conference on Extending Database Technology Avignon, France, March 25–29, 1996 Proceedings 5. Springer Berlin Heidelberg, 1996.

44. Hernández-Calderón, José Guillermo et al. A System to Match Behaviors and Performance of Learners From User-Object Interactions: Model and Architecture. International Journal of Information Technologies and Systems Approach (IJITSA) 12.2 (2019): 82-103.

45. Magdalena Cantabella et al. Analysis of student behavior in learning management systems through a Big Data framework. Future Generation Computer Systems 90 (2019): 262-272.

46. José Antonio Xohua-Chacón, Edgard Iván Benítez-Guerrero, Carmen Mezura-Godoy. A tangible system for learning relational algebra. Revista Colombiana de Computación 19.1 (2018): 39-55.

47. United States. National Commission for the Protection of Human Subjects of Biomedical, and Behavioral Research. The Belmont report: ethical principles and guidelines for the protection of human subjects of research. Vol. 1. Department of Health, Education, and Welfare, National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, 1978.

48. Krisztian Buza, Lars Schmidt-Thieme. Motif-based classification of time series with bayesian networks and svms. Advances in Data Analysis, Data Handling and Business Intelligence: Proceedings of the 32nd Annual Conference of the Gesellschaft für Klassifikation eV, Joint Conference with the British Classification Society (BCS) and the Dutch/Flemish Classification Society (VOC), Helmut-Schmidt-University, Hamburg, July 16-18, 2008. Springer Berlin Heidelberg, 2010.

49. Jia-Min Ren, Jyh-Shing Roger Jang. Time-constrained sequential pattern discovery for music genre classification. 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2011.

50. Vanessa Alejandra Camacho Vázquez. Detección de emociones negativas en las redes sociales., Ph.D. Thesis, 2017.

51. Samantha Jiménez, R. Juárez-Ramírez, Víctor H Castillo, Alan Ramírez Noriega. The Role of Personality in Motivation to use an Affective Feedback System. Program Comput Soft 47, 793–802 (2021). https://doi.org/10.1134/S0361768821080156.

52. J. R. Aguilar-Cisneros, R. Valerdi, B. P. Sullivan. Students’ Systems Thinking Competency Level Detection through Software Cost Estimation Concept Modeling. Program Comput Soft 48, 499–512 (2022). https://doi.org/10.1134/S0361768822080060.

53. Kochegurova E. A., Martynova Y. A. Aspects of Continuous User Identification Based on Free Texts and Hidden Monitoring. Program Comput Soft 46, 12–24 (2020). https://doi.org/10.1134/S036176882001003X.

54. H. Del Ángel-Flores, E. López-Domínguez, Y. Hernández-Velázquez et al. Usability Evaluation of a Mobile Learning Platform Focused on Learning Monitoring and Customization based on a Laboratory Study. Program Comput Soft 48, 583–597 (2022). https://doi.org/10.1134/S0361768822080102.