From Interaction Data to Personalized Learning: Mining User-Object Interactions in Intelligent Environments
José-Guillermo HERNÁNDEZ-CALDERÓN,
Edgard Ivan BENÍTEZ-GUERRERO,
José Rafael ROJANO-CÁCERES,
Carmen MEZURA-GODOY https://doi.org/10.15514/ISPRAS-2024-36(1)-10
Abstract
The aim of this work is to contribute to the personalization of intelligent learning environments by analyzing user-object interaction data to identify On-Task and Off-Task behaviors. This is accomplished by monitoring and analyzing users' interactions while performing academic activities with a tangible-intangible hybrid system in a university intelligent environment configuration. With the proposal of a framework and the Orange Data Mining tool and the Neural Network, Random Forest, Naive Bayes, and Tree classification models, training and testing was carried out with the user-object interaction records of the 13 students (11 for training and two for testing) to identify representative sequences of behavior from user-object interaction records. The two models that had the best results, despite the small number of data, were the Neural Network and Naive Bayes. Although a more significant amount of data is necessary to perform a classification adequately, the process allowed exemplifying this process so that it can later be fully incorporated into an intelligent educational system to contribute to build personalized environments.
Keywords
intelligent learning environments,
user-behavior identification,
user-object interaction data,
data mining.
Supporting agencies: The first author gratefully acknowledges CONAHCYT for scholarship No. 421557 for graduate studies.
About the Authors
José-Guillermo HERNÁNDEZ-CALDERÓN
Facultad de Estadística e Informática, Universidad Veracruzana
Mexico
Mexico
Ph. D. in Computer Science from the University of Veracruz in México. Professor at the Faculty of Statistics and Informatics of the University of Veracruz in Mexico. Research interests: Human Computer Interaction, Artificial Intelligence, Collaborative Computing, and Visualization.
Edgard Ivan BENÍTEZ-GUERRERO
Facultad de Estadística e Informática, Universidad Veracruzana
Mexico
Mexico
Ph. D. in Computer Science from the University of Grenoble in France. Professor at the Faculty of Statistics and Informatics of the University of Veracruz in Mexico. Research interests: Human Computer Interaction, Artificial Intelligence, Collaborative Computing, Data Management and Visualization.
José Rafael ROJANO-CÁCERES
Facultad de Estadística e Informática, Universidad Veracruzana
Mexico
Mexico
PhD in computer science from the Instituto Tecnológico de Estudios Superiores Monterrey campus Cuernavaca. Master in Artificial Intelligence from the Universidad Veracruzana. Currently Full Time Professor at the Faculty of Statistics and Informatics of the Universidad Veracruzana. His research areas include semantic web, collaborative tools, and disability.
Carmen MEZURA-GODOY
Facultad de Estadística e Informática, Universidad Veracruzana
Mexico
Mexico
Carmen MEZURA-GODOY – PhD in Computer Science from the University of Savoie in France. Professor at the Faculty of Statistics and Informatics of the University of Veracruz in Mexico. Main research interests: Human-Computer Interaction, User eXperience-UX, Computer Support Collaborative Work, Visualization and Multiagent Systems.
References
1. G. Virone et al., Behavioral Patterns of Older Adults in Assisted Living, in IEEE Transactions on Information Technology in Biomedicine, vol. 12, no. 3, pp. 387-398, May 2008, doi: 10.1109/TITB.2007.904157.
2. Pantic Maja, et al. Human computing and machine understanding of human behavior: A survey. Proceedings of the 8th international conference on Multimodal Interfaces. 2006.
3. Favela Jesus. Activity, Behavior, and Context: The ABC of Pervasive Healthcare Research. Intelligent Environments (Workshops). 2012.
4. Nef Tobias et al. Evaluation of three state-of-the-art classifiers for recognition of activities of daily living from smart home ambient data. Sensors 15.5 (2012): 11725-11740.
5. Vinh La The, et al. Semi-Markov conditional random fields for accelerometer-based activity recognition. Applied Intelligence 35 (2011): 226-241.
6. Moore Dennis W. et al. Increasing on-task behavior in students in a regular classroom: Effectiveness of a self-management procedure using a tactile prompt. Journal of Behavioral Education 22 (2013): 302-311.
7. Godwin Karrie E. et al. Off-task behavior in elementary school children. Learning and Instruction 44 (2016): 128-143.
8. Doddannara Lakshmi S. et al. Exploring the relationships between design, students' affective states, and disengaged behaviors within an ITS. Artificial Intelligence in Education: 16th International Conference, AIED 2013, Memphis, TN, USA, July 9-13, 2013. Proceedings 16. Springer Berlin Heidelberg, 2013.
9. Wang Li‐Chun, Ming‐Puu Chen. The effects of game strategy and preference‐matching on flow experience and programming performance in game‐based learning. Innovations in Education and Teaching International 47.1 (2010): 39-52.
10. Schaufeli Wilmar B., Arnold Bakker. UWES Utrecht work engagement scale preliminary manual. Occupational Health Psychology Unit (2003).
11. Bumbacco C., Scharfe E. (2023). Why attachment matters: first-year post-secondary students' experience of burnout, dIsengagement, And drop-out. Journal of College Student Retention: Research, Theory & Practice, 24(4), 988-1001.
12. Zhang J., Mohd Yunos Z, Haron H. Interactivity Recognition Graph Neural Network (IR-GNN) Model for Improving Human–Object Interaction Detection. Electronics. 2023; 12(2):470. https://doi.org/10.3390/electronics12020470
13. Qing Ye, Xiuju Xu, Human-object interaction detection based on graph model, Proc. SPIE 12610, Third International Conference on Artificial Intelligence and Computer Engineering (ICAICE 2022), 126100A (28 April 2023); https://doi.org/10.1117/12.2671248
14. Mazzamuto Michele, et al. A wearable device application for human-object interactions detection. International Conference on Computer Vision Theory and Applications (VISAPP). 2023.
15. Luo Tianlun, et al. From detection to understanding: A survey on representation learning for human-object interaction. Neurocomputing (2023): 126243.
16. Kim, Sanghyun, Deunsol Jung, Minsu Cho. Relational Context Learning for Human-Object Interaction Detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
17. Ye Qing, et al. Human object interaction detection based on feature optimization and key human-object enhancement. Journal of Visual Communication and Image Representation 93 (2023): 103824.
18. Zidianakis Emmanouil, et al. Building a sensory infrastructure to support interaction and monitoring in ambient intelligence environments. Distributed, Ambient, and Pervasive Interactions: Second International Conference, DAPI 2014, Held as Part of HCI Interational 2014, Heraklion, Crete, Greece, June 22-27, 2014. Proceedings 2. Springer International Publishing, 2014.
19. D'Mello Sidney K. et al. Multimodal capture of teacher-student interactions for automated dialogic analysis in live classrooms. Proceedings of the 2015 ACM on international conference on multimodal interaction. 2015.
20. Belapurkar Neha, Sagar Shelke, Baris Aksanli. The case for ambient sensing for human activity detection. Proceedings of the 8th International Conference on the Internet of Things. 2018.
21. Jayarajah Kasthuri et al. Need accurate user behaviour? pay attention to groups!. Proceedings of the 2015 ACM international joint conference on pervasive and ubiquitous computing. 2015.
22. Baker Ryan Shaun et al. Off-task behavior in the cognitive tutor classroom: When students "Game the system". Proceedings of the SIGCHI conference on Human factors in computing systems. 2004.
23. Baker Ryan. Modeling and understanding students' off-task behavior in intelligent tutoring systems. Proceedings of the SIGCHI conference on Human factors in computing systems. 2007.
24. Leonidis Asterios et al. An intelligent task assignment and personalization system for students' online collaboration. Universal Access in Human-Computer Interaction. Applications and Services: 6th International Conference, UAHCI 2011, Held as Part of HCI International 2011, Orlando, FL, USA, July 9-14, 2011, Proceedings, Part IV 6. Springer Berlin Heidelberg, 2011.
25. Sanish Rai, Xiaolin Hu. Behavior pattern detection for data assimilation in agent-based simulation of smart environments. 2013 IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT). Vol. 2. IEEE, 2013.
26. Wang Qunbo, Wenjun Wu, Yuxing Qi. A Learning Analytic Model for Smart Classroom. Web and Big Data: APWeb-WAIM 2018 International Workshops: MWDA, BAH, KGMA, DMMOOC, DS, Macau, China, July 23–25, 2018, Revised Selected Papers 2. Springer International Publishing, 2018.
27. Korozi Maria et al. LECTOR: towards reengaging students in the educational process inside smart classrooms. Intelligent Human Computer Interaction: 9th International Conference, IHCI 2017, Evry, France, December 11-13, 2017, Proceedings 9. Springer International Publishing, 2017.
28. White Ryen W., Ahmed Hassan Awadallah, Robert Sim. Task completion detection: A study in the context of intelligent systems. Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2019.
29. Montebello Matthew. Assisting education through real-time learner analytics. 2018 IEEE Frontiers in Education Conference (FIE). IEEE, 2018.
30. Matsui Kanae, Tatsuhiko Kasai, Keiya Sakai. Challenges for data collecting of teacher and student'behavior in different types of class using video and wearable device. 2019 Joint 8th International Conference on Informatics, Electronics & Vision (ICIEV) and 2019 3rd International Conference on Imaging, Vision & Pattern Recognition (icIVPR). IEEE, 2019.
31. Prabono Aria Ghora, Seok-Lyong Lee, Bernardo Nugroho Yahya. Context-based similarity measure on human behavior pattern analysis. Soft Computing 23 (2019): 5455-5467.
32. Biswas Gautam, Brian Sulcer. Visual exploratory data analysis methods to characterize student progress in intelligent learning environments. 2010 International Conference on Technology for Education. IEEE, 2010.
33. G. Mathioudakis et al. Amiria: Real-time teacher assistance tool for an ambient intelligence classroom. Proceedings of the Fifth International Conference on Mobile, Hybrid, and Online Learning (eLmL 2013). 2013.
34. Barbara Bruno et al. Analysis of human behavior recognition algorithms based on acceleration data. 2013 IEEE International Conference on Robotics and Automation. IEEE, 2013.
35. Valérian Guivarch et al. Hybrid system to analyze user's behaviour. 2016 IEEE Symposium Series on Computational Intelligence (SSCI). IEEE, 2016.
36. Louis Atallah, Guang-Zhong Yang. The use of pervasive sensing for behaviour profiling – a survey. Pervasive and mobile computing 5.5 (2009): 447-464.
37. Zacharoula Papamitsiou, Eirini Karapistoli, Anastasios A. Economides. Applying classification techniques on temporal trace data for shaping student behavior models. Proceedings of the sixth international conference on learning analytics & knowledge. 2016.
38. Valeria Soto-Mendoza, et al. Detecting abnormal behaviours of institutionalized older adults through a hybrid-inference approach. Pervasive and Mobile Computing 40 (2017): 708-723.
39. Andreas D. Lattner, et al. Sequential pattern mining for situation and behavior prediction in simulated robotic soccer. RoboCup 2005: Robot Soccer World Cup IX 9. Springer Berlin Heidelberg, 2006.
40. Magnus S. Magnusson. Discovering hidden time patterns in behavior: T-patterns and their detection. Behavior research methods, instruments, & computers 32.1 (2000): 93-110.
41. Maurizio Casarrubea et al. T-pattern analysis for the study of temporal structure of animal and human behavior: a comprehensive review. Journal of neuroscience methods 239 (2015): 34-46.
42. Carlos Santoyo et al. Observational analysis of the organization of on-task behavior in the classroom using complementary data analyses. Anales de Psicología, 2017, vol. 33, num. 3, p. 497-514 (2017).
43. Ramakrishnan Srikant, Rakesh Agrawal. Mining sequential patterns: Generalizations and performance improvements. Advances in Database Technology—EDBT'96: 5th International Conference on Extending Database Technology Avignon, France, March 25–29, 1996 Proceedings 5. Springer Berlin Heidelberg, 1996.
44. Hernández-Calderón, José Guillermo et al. A System to Match Behaviors and Performance of Learners From User-Object Interactions: Model and Architecture. International Journal of Information Technologies and Systems Approach (IJITSA) 12.2 (2019): 82-103.
45. Magdalena Cantabella et al. Analysis of student behavior in learning management systems through a Big Data framework. Future Generation Computer Systems 90 (2019): 262-272.
46. José Antonio Xohua-Chacón, Edgard Iván Benítez-Guerrero, Carmen Mezura-Godoy. A tangible system for learning relational algebra. Revista Colombiana de Computación 19.1 (2018): 39-55.
47. United States. National Commission for the Protection of Human Subjects of Biomedical, and Behavioral Research. The Belmont report: ethical principles and guidelines for the protection of human subjects of research. Vol. 1. Department of Health, Education, and Welfare, National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research, 1978.
48. Krisztian Buza, Lars Schmidt-Thieme. Motif-based classification of time series with bayesian networks and svms. Advances in Data Analysis, Data Handling and Business Intelligence: Proceedings of the 32nd Annual Conference of the Gesellschaft für Klassifikation eV, Joint Conference with the British Classification Society (BCS) and the Dutch/Flemish Classification Society (VOC), Helmut-Schmidt-University, Hamburg, July 16-18, 2008. Springer Berlin Heidelberg, 2010.
49. Jia-Min Ren, Jyh-Shing Roger Jang. Time-constrained sequential pattern discovery for music genre classification. 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2011.
50. Vanessa Alejandra Camacho Vázquez. Detección de emociones negativas en las redes sociales., Ph.D. Thesis, 2017.
51. Samantha Jiménez, R. Juárez-Ramírez, Víctor H Castillo, Alan Ramírez Noriega. The Role of Personality in Motivation to use an Affective Feedback System. Program Comput Soft 47, 793–802 (2021). https://doi.org/10.1134/S0361768821080156.
52. J. R. Aguilar-Cisneros, R. Valerdi, B. P. Sullivan. Students’ Systems Thinking Competency Level Detection through Software Cost Estimation Concept Modeling. Program Comput Soft 48, 499–512 (2022). https://doi.org/10.1134/S0361768822080060.
53. Kochegurova E. A., Martynova Y. A. Aspects of Continuous User Identification Based on Free Texts and Hidden Monitoring. Program Comput Soft 46, 12–24 (2020). https://doi.org/10.1134/S036176882001003X.
54. H. Del Ángel-Flores, E. López-Domínguez, Y. Hernández-Velázquez et al. Usability Evaluation of a Mobile Learning Platform Focused on Learning Monitoring and Customization based on a Laboratory Study. Program Comput Soft 48, 583–597 (2022). https://doi.org/10.1134/S0361768822080102.
Review
For citations:
HERNÁNDEZ-CALDERÓN J., BENÍTEZ-GUERRERO E.I., ROJANO-CÁCERES J.R., MEZURA-GODOY C. From Interaction Data to Personalized Learning: Mining User-Object Interactions in Intelligent Environments. Proceedings of the Institute for System Programming of the RAS (Proceedings of ISP RAS). 2024;36(1):157-174. https://doi.org/10.15514/ISPRAS-2024-36(1)-10
Email this article 