Apparent predictability of learning-oriented participation from behavioral and contextual student data: Near-duplicate leakage and the roles of discussion participation and motivation
Learning analytics increasingly predicts student participation from behavioral and contextual data, but performance can be overstated when outcome components enter the predictors, related records appear across cross-validation splits, or uncertainty is ignored. This study examined whether learning-oriented participation can be predicted from a publicly available dataset and whether discussion participation and motivation add incremental information. Of 14,003 records (16 variables), 1,534 were exact duplicates that could not plausibly arise by chance (expected coincidental pairs 3.8 × 10−5). Of the 12,469 unique records, 10,701 (85.8%) belonged to multi-record near-duplicate families, within which only Resources, Motivation, Extracurricular, and Internet. A formative Learning-Oriented Participation Index (LOPI) averaged percentile-ranked Attendance, AssignmentCompletion, and OnlineCourses. Elastic Net, Extra Trees, and CatBoost were evaluated with nested five-fold cross-validation, keeping each family within one fold. Under row-wise validation, Extra Trees appeared to reduce mean absolute error (MAE) by 0.305 points, and Discussions appeared informative. Under family-grouped validation, no model outperformed the mean baseline (Extra Trees MAE = 13.663, R2 = −0.011; baseline: 13.605), the Discussions gain disappeared (ΔMAE = −0.017, 95% confidence interval = −0.035 to 0.002), Motivation added no stable information, the Discussions × Motivation interaction was unsupported, and lowest-quartile identification was at chance (area under the receiver operating characteristic curve: 0.46). Split-conformal intervals achieved near-nominal coverage (89.2% and 94.5% for nominal 90% and 95%) but spanned 55.1 and 64.3 LOPI points. The apparent predictability reflected near-duplicate leakage; public educational datasets should be audited for duplicate structure and evaluated with grouped validation before interpretation.
Abukader, A., Alzu'bi, A., & Adegboye, O. R. (2025). Intelligent system for student performance prediction: An educational data mining approach using metaheuristic-optimized LightGBM with SHAP-based learning analytics. Applied Sciences, 15, 10875. https://doi.org/10.3390/app152010875
Akpabio, I. I., & Savari, S. A. (2021). Uncertainty quantification of machine learning models: On conformal prediction. Journal of Micro/Nanopatterning, Materials, and Metrology, 20(4), 041206. https://doi.org/10.1117/1.JMM.20.4.041206
Arada, B. (2025). Level of social presence, social interaction, collaborative learning, and satisfaction in a BAMS course Discord server. International Journal on Open and Distance e-Learning, 11(1). https://doi.org/10.58887/ijodel.v11i1.294
Azuela, L., Azuela, H. P., Felipe, A. L., Aragoncillo, A., & Eleazar, M. (2025). From data to action: Narrative benchmarking and comparative evaluation of explainable AI-driven supervised machine learning models for academic risk prediction. In 2025 IEEE International Conference on Emerging Trends in Engineering and Computing (ETECOM). IEEE. https://doi.org/10.1109/etecom66111.2025.11319004
Choi, W. C., Lam, C.-T., & Mendes, A. J. (2024). Analyzing the interpretability of machine learning prediction on student performance using SHapley additive exPlanations. In Proceedings of TALE 2024. IEEE. https://doi.org/10.1109/tale62452.2024.10834292
Deutschmann, N., & Rigotti, M. (2023). Adaptive conformal regression with Jackknife+ rescaled scores. arXiv. https://doi.org/10.48550/arXiv.2305.19901
Diamant, N., Hajiramezanali, E., Biancalani, T., & Scalia, G. (2023). Conformalized deep splines for optimal and efficient prediction sets. arXiv. https://doi.org/10.48550/arxiv.2311.00774
Diamantopoulos, A., & Winklhofer, H. M. (2001). Index construction with formative indicators: An alternative to scale development. Journal of Marketing Research, 38(2), 269–277. https://doi.org/10.1509/jmkr.38.2.269.18845
Eccles, J. S., & Wigfield, A. (2002). Motivational beliefs, values, and goals. Annual Review of Psychology, 53, 109–132. https://doi.org/10.1146/annurev.psych.53.100901.135153
Fisher, A., Rudin, C., & Dominici, F. (2019). All models are wrong, but many are useful: Learning a variable's importance by studying an entire class of prediction models simultaneously. Journal of Machine Learning Research, 20, 177.
Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential of the concept, state of the evidence. Review of Educational Research, 74(1), 59–109. https://doi.org/10.3102/00346543074001059
George, E. B., Senthilkumar, R., Al-Junaibi, F., & Al-Shuaibi, Z. (2025). Explainable AI methods for predicting student grades and improving academic success. Journal of Information Systems Engineering and Management, 10. https://doi.org/10.52783/jisem.v10i23s.3680
Getenet, S., Tualaulelei, E., Singh, N., Pillay, Y., & Hammermeister, K.-A. (2026). Interactive technologies and student engagement in online higher education: A multidimensional analysis. Technology, Pedagogy and Education, 35(1), 83–101. https://doi.org/10.1080/1475939X.2025.2498483
Gharbaoui, H., Mansouri, K., & Poirier, F. (2024). Improving student engagement and success in computer programming courses through social learning in online environments. International Journal of Engineering Pedagogy, 14(6), 54–68. https://doi.org/10.3991/ijep.v14i6.48705
Hajibabaee, P., Pourkamali-Anaraki, F., & Hariri-Ardebili, M. (2024). Adaptive conformal prediction intervals using data-dependent weights with application to seismic response prediction. IEEE Access, 12, 53579–53597. https://doi.org/10.1109/ACCESS.2024.3387858
Jarvis, C. B., MacKenzie, S. B., & Podsakoff, P. M. (2003). A critical review of construct indicators and measurement model misspecification in marketing and consumer research. Journal of Consumer Research, 30(2), 199–218. https://doi.org/10.1086/376806
Kahu, E. R. (2013). Framing student engagement in higher education. Studies in Higher Education, 38(5), 758–773. https://doi.org/10.1080/03075079.2011.598505
Kamaruddin, S. F., Saifullizam, N. Z., Mohd Zahid, A. Z., Khalid, N., & Mat Zin, M. (2025). Exploring the influence of learner-to-learner interaction in online learning. International Journal of Research and Innovation in Social Science. https://doi.org/10.47772/IJRISS.2025.90900059
Kapoor, S., & Narayanan, A. (2023). Leakage and the reproducibility crisis in machine-learning-based science. Patterns, 4(9), 100804. https://doi.org/10.1016/j.patter.2023.100804
Kumar, B., Kumar, T., Nadakuditi, S., Patel, H., & Gupta, K. (2024). Comparing conformal and quantile regression for uncertainty quantification: An empirical investigation. International Journal of Computing and Engineering, 5(5), 1–8. https://doi.org/10.47941/ijce.1925
Kuzilek, J., Hlosta, M., & Zdrahal, Z. (2017). Open University Learning Analytics dataset. Scientific Data, 4, 170171. https://doi.org/10.1038/sdata.2017.171
Lei, J., G'Sell, M., Rinaldo, A., Tibshirani, R. J., & Wasserman, L. (2018). Distribution-free predictive inference for regression. Journal of the American Statistical Association, 113(523), 1094–1111. https://doi.org/10.1080/01621459.2017.1307116
Lin, X., & Sun, Q. (2024). Discussion activities in asynchronous online learning: Motivating adult learners' interactions. The Journal of Continuing Higher Education, 72(1), 84–103. https://doi.org/10.1080/07377363.2022.2119803
Liu, Z., Zhou, X., & Liu, Y. (2025). Student dropout prediction using ensemble learning with SHAP-based explainable AI analysis. Journal of Social System and Policy Analysis, 2(3), 111–132. https://doi.org/10.62762/jsspa.2025.321501
Livieris, I. E., Karacapilidis, N., Domalis, G., & Tsakalidis, D. (2023). An advanced explainable and interpretable ML-based framework for educational data mining. In Artificial intelligence applications and innovations. Springer, Cham, Switzerland. https://doi.org/10.1007/978-3-031-42134-1_9
Lundberg, S. M., & Lee, S.-I. (2017). A unified approach to interpreting model predictions. Advances in Neural Information Processing Systems, 30, 4765–4774.
Luo, R., & Zhou, Z. (2025). Conformal thresholded intervals for efficient regression. Proceedings of the AAAI Conference on Artificial Intelligence, 39(18), 19216–19223. https://doi.org/10.1609/aaai.v39i18.34115
Marchi, E. J., & Liebl, M. (2026). U-TraCE: A conformal prediction approach to uncertainty quantification in black-box models. Machine Learning: Science and Technology, 7(1), 015017. https://doi.org/10.1088/2632-2153/ae35ce
Mary, S. P., Gambhir, V., Mahalakshmi, R., Begum, B., Kala, & Mahajan, S. (2025). Optimizing student engagement prediction with machine learning and big data analytics. In Proceedings of ICMCTC 2025. IEEE. https://doi.org/10.1109/ICMCTC62214.2025.11196115
Mudau, P. K., & van den Berg, G. (2023). Guidelines for supporting a community of inquiry through graded online discussion forums in higher education. Education Sciences, 13(9), 963. https://doi.org/10.3390/educsci13090963
Murattaly, Y., & Serek, A. (2025). Predictive analytics for student engagement in e-learning systems. Journal of Emerging Technologies and Computing, 3(3), 23–30. https://doi.org/10.47344/88c0mc52
Navaneetha, T., Saravanakumar, R., Vini Infanta, A., Mohammed, M. A., Sumra, V., & Ali, M. (2025). Behavioral analytics in e-learning: Deep learning for real-time interaction and performance prediction. In Proceedings of ICETETSIP 2025. IEEE. https://doi.org/10.1109/ICETETSIP64213.2025.11156864
Nnadi, L. C., Watanobe, Y., Rahman, M. M., & John-Otumu, A. M. (2024). Prediction of students' adaptability using explainable AI in educational machine learning models. Applied Sciences, 14(12), 5141. https://doi.org/10.3390/app14125141
Onwuzurike, M. A., & Kpogli, S. A. (2025). Predictive modeling of student engagement and behavioral outcomes using machine learning techniques in technology-enhanced classrooms. International Journal of Scientific Research in Humanities and Social Sciences, 2(6), 58–79. https://doi.org/10.32628/ijsrhss2525135
Pashler, H., McDaniel, M., Rohrer, D., & Bjork, R. (2008). Learning styles: Concepts and evidence. Psychological Science in the Public Interest, 9(3), 105–119. https://doi.org/10.1111/j.1539-6053.2009.01038.x
Pintrich, P. R., Smith, D. A. F., Garcia, T., & McKeachie, W. J. (1991). A manual for the use of the Motivated Strategies for Learning Questionnaire (MSLQ). National Center for Research to Improve Postsecondary Teaching and Learning, Ann Arbor, MI. https://files.eric.ed.gov/fulltext/ED338122.pdf
Rachha, A., & Seyam, M. (2023). Explainable AI in education: Current trends, challenges, and opportunities. In Proceedings of IEEE SoutheastCon 2023. IEEE. https://doi.org/10.1109/SoutheastCon51012.2023.10115140
Roberts, D. R., Bahn, V., Ciuti, S., Boyce, M. S., Elith, J., Guillera-Arroita, G., Hauenstein, S., Lahoz-Monfort, J. J., Schröder, B., Thuiller, W., Warton, D. I., Wintle, B. A., Hartig, F., & Dormann, C. F. (2017). Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure. Ecography, 40(8), 913–929. https://doi.org/10.1111/ecog.02881
Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist, 55(1), 68–78. https://doi.org/10.1037/0003-066X.55.1.68
Sheeran, P. (2002). Intention–behavior relations: A conceptual and empirical review. European Review of Social Psychology, 12(1), 1–36. https://doi.org/10.1080/14792772143000003
Shin, A. (2023). Exploring the role of interaction in engagement and satisfaction within virtual learning environments. Journal of Student Research, 12(3). https://doi.org/10.47611/jsrhs.v12i3.5014
Singh, G., Moncrieff, G., Venter, Z., Cawse-Nicholson, K., Slingsby, J., & Robinson, T. B. (2024). Uncertainty quantification for probabilistic machine learning in earth observation using conformal prediction. Scientific Reports, 14, 16166. https://doi.org/10.1038/s41598-024-65954-w
Slade, S., & Prinsloo, P. (2013). Learning analytics: Ethical issues and dilemmas. American Behavioral Scientist, 57(10), 1510–1529. https://doi.org/10.1177/0002764213479366
Sun, S. (2022). Conformal methods for quantifying uncertainty in spatiotemporal data: A survey. arXiv. https://doi.org/10.48550/arXiv.2209.03580
Tirumanadham, N. S. K. M. K., Thaiyalnayaki, S., & Ganesan, V. (2025). Towards smarter e-learning: Real-time analytics and machine learning for personalized education. International Journal of Computational and Experimental Science and Engineering, 11(1). https://doi.org/10.22399/ijcesen.786
Triantafyllou, S. A., & Sapounidis, T. (2024). Algorithmic approaches to the Traveling Salesman Problem for the development of students' computational thinking. In Y. Farhaoui (Ed.), Artificial intelligence, big data, IOT and block chain in healthcare: From concepts to applications (pp. 509–523). Springer, Cham, Switzerland. https://doi.org/10.1007/978-3-031-65018-5_48
Varma, S., & Simon, R. (2006). Bias in error estimation when using cross-validation for model selection. BMC Bioinformatics, 7, 91. https://doi.org/10.1186/1471-2105-7-91
Wanjale, K., Suryawanshi, A., Mathurkar, P., Raut, A., & Pawar, Y. (2025). A data-driven approach to personalized learning and course enhancement in education technology. In Proceedings of WorldS4 2025. IEEE. https://doi.org/10.1109/worldsuas66815.2025.11198927
Xu, Y., Guo, W., & Wei, Z. (2025). Selective conformal risk control. arXiv. https://doi.org/10.48550/arxiv.2512.12844
Yu, J., Wang, Y., Zhong, Q., Luo, G., Mao, Y., Sun, K., Feng, W., Xu, W., Cao, S., Zeng, K., Yao, Z., Hou, L., Lin, Y., Li, P., Zhou, J., Xu, B., Li, J., Tang, J., & Sun, M. (2021). MOOCCubeX: A large knowledge-centered repository for adaptive learning in MOOCs. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management (pp. 4643–4652). Association for Computing Machinery, New York, NY. https://doi.org/10.1145/3459637.3482010
Zhang, Y., & Liu, Y.-K. (2025). Conformal prediction beyond the horizon: Distribution-free inference for policy evaluation. arXiv. https://doi.org/10.48550/arxiv.2510.26026
Zhou, X., Li, Q., Xu, D., Holton, A., & Sato, B. (2023). The promise of using study-together groups to promote engagement and performance in online courses: Experimental evidence on academic and non-cognitive outcomes. The Internet and Higher Education. https://doi.org/10.1016/j.iheduc.2023.100922
