Knowledge-aware personalized next-item recommendation for Chinese online education: Evaluating pretrained RoBERTa semantic embeddings
Personalized learning systems require recommendations reflecting learner history, knowledge state, and observed instructional sequence. This study evaluates a knowledge-aware next-item ranking framework for Chinese online education using XES3G5M. The analysis included 14,453 development students with 4,446,374 question-level interactions and an independent test set of 3,613 students with 1,102,810 interactions. Student state was represented through Bayesian knowledge-component mastery, response history, concept transitions, item difficulty, and 768-dimensional robustly optimized bidirectional encoder representations from transformers pretraining approach (RoBERTa)-derived semantic embeddings supplied with the dataset; no external generative language model was used during training or inference. Under a 100-candidate sampled evaluation, the tuned linear hybrid achieved hit rate at 10 = 0.779, normalized discounted cumulative gain at 10 (NDCG@10) = 0.459, and mean reciprocal rank = 0.372. Modern sequential baselines were stronger, with self-attentive sequential recommendation reaching NDCG@10 = 0.584 and gated recurrent unit-based session-based recommendation (GRU4Rec) reaching 0.729, while an extreme gradient boosting pairwise ranker using the same five engineered features reached 0.486. Development-fold tuning assigned zero weight to semantic similarity and difficulty fit, and the no-semantic ablation produced identical test rankings. Full-catalog ranking over all 7,652 metadata questions reduced the linear hybrid to NDCG@10 = 0.140, while GRU4Rec reached 0.271. The evidence supports the importance of sequence and platform-exposure structure in reproducing the next observed question, but shows no incremental benefit from supplied RoBERTa embeddings under the present linear specification. A platform-structure diagnostic showed popularity plus transition nearly matched the full linear model, and the primary evaluation (≥ 160 prior interactions per learner) is not cold-start evidence. The target is next-observed-item fidelity in this particular evaluation setting, not a pedagogically optimal path, and findings are limited to one Chinese K-12 mathematics platform.
Abdelrahman, G., Wang, Q., & Nunes, B. P. (2023). Knowledge tracing: A survey. ACM Computing Surveys, 55(11), Article 224. https://doi.org/10.1145/3569576
Bond, M., Khosravi, H., De Laat, M., Bergdahl, N., Negrea, V., Oxley, E., Pham, P., Chong, S. W., & Siemens, G. (2024). A meta systematic review of artificial intelligence in higher education: A call for increased ethics, collaboration, and rigour. International Journal of Educational Technology in Higher Education, 21(1), 4. https://doi.org/10.1186/s41239-023-00436-z
Chan, C. K. Y. (2023). A comprehensive AI policy education framework for university teaching and learning. International Journal of Educational Technology in Higher Education, 20(1), 38. https://doi.org/10.1186/s41239-023-00408-3
Chen, L., Chen, P., & Lin, Z. (2020). Artificial intelligence in education: A review. IEEE Access, 8, 75264–75278. https://doi.org/10.1109/ACCESS.2020.2988510
Chen, T., & Guestrin, C. (2016). XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 785–794). Association for Computing Machinery. https://doi.org/10.1145/2939672.2939785
Choi, Y., Lee, Y., Cho, J., Baek, J., Kim, B., Cha, Y., Shin, D., Bae, C., & Heo, J. (2020). Towards an appropriate query, key, and value computation for knowledge tracing. In Proceedings of the Seventh ACM Conference on Learning @ Scale (pp. 341–344). Association for Computing Machinery. https://doi.org/10.1145/3386527.3405945
Choi, Y., Lee, Y., Shin, D., Cho, J., Park, S., Lee, S., Baek, J., Bae, C., Kim, B., & Heo, J. (2020). EdNet: A large-scale hierarchical dataset in education. In: Artificial Intelligence in Education: AIED 2020 (Lecture Notes in Computer Science, Vol. 12164, pp. 69–73). Heidelberg, Germany: Springer. https://doi.org/10.1007/978-3-030-52240-7_13
Cotton, D. R. E., Cotton, P. A., & Shipway, J. R. (2024). Chatting and cheating: Ensuring academic integrity in the era of ChatGPT. Innovations in Education and Teaching International, 61(2), 228–239. https://doi.org/10.1080/14703297.2023.2190148
Crompton, H., & Burke, D. (2023). Artificial intelligence in higher education: The state of the field. International Journal of Educational Technology in Higher Education, 20(1), 22. https://doi.org/10.1186/s41239-023-00392-8
Dwivedi, Y. K., Kshetri, N., Hughes, L., Slade, E. L., Jeyaraj, A., Kar, A. K., Baabdullah, A. M., Koohang, A., Raghavan, V., Ahuja, M., Albanna, H., Albashrawi, M. A., Al-Busaidi, A. S., Balakrishnan, J., Barlette, Y., Basu, S., Bose, I., Brooks, L., Buhalis, D., … Wright, R. (2023). So what if ChatGPT wrote it? Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy. International Journal of Information Management, 71, 102642. https://doi.org/10.1016/j.ijinfomgt.2023.102642
Farrokhnia, M., Banihashem, S. K., Noroozi, O., & Wals, A. (2024). A SWOT analysis of ChatGPT: Implications for educational practice and research. Innovations in Education and Teaching International, 61(3), 460–474. https://doi.org/10.1080/14703297.2023.2195846
Ghosh, A., Heffernan, N., & Lan, A. S. (2020). Context-aware attentive knowledge tracing. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (pp. 2330–2339). Association for Computing Machinery. https://doi.org/10.1145/3394486.3403282
Grassini, S. (2023). Shaping the future of education: Exploring the potential and consequences of AI and ChatGPT in educational settings. Education Sciences, 13(7), 692. https://doi.org/10.3390/educsci13070692
He, X., Deng, K., Wang, X., Li, Y., Zhang, Y., & Wang, M. (2020). LightGCN: Simplifying and powering graph convolution network for recommendation. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 639–648). Association for Computing Machinery. https://doi.org/10.1145/3397271.3401063
Hidasi, B., Karatzoglou, A., Baltrunas, L., & Tikk, D. (2015). Session-based Recommendations with Recurrent Neural Networks (Version 4). arXiv. arXiv:1511.06939. https://doi.org/10.48550/ARXIV.1511.06939
Hwang, G. J., Xie, H., Wah, B. W., & Gašević, D. (2020). Vision, challenges, roles and research issues of artificial intelligence in education. Computers and Education: Artificial Intelligence, 1, 100001. https://doi.org/10.1016/j.caeai.2020.100001
Kang, W. C., & McAuley, J. (2018). Self-attentive sequential recommendation. In Proceedings of the 2018 IEEE International Conference on Data Mining (ICDM) (pp. 197–206). IEEE. https://doi.org/10.1109/ICDM.2018.00035
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., … Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103, 102274. https://doi.org/10.1016/j.lindif.2023.102274
Khosravi, H., Shum, S. B., Chen, G., Conati, C., Tsai, Y. S., Kay, J., Knight, S., Martinez-Maldonado, R., Sadiq, S., & Gašević, D. (2022). Explainable artificial intelligence in education. Computers and Education: Artificial Intelligence, 3, 100074. https://doi.org/10.1016/j.caeai.2022.100074
Lim, W. M., Gunasekara, A., Pallant, J. L., Pallant, J. I., & Pechenkina, E. (2023). Generative AI and the future of education: Ragnarök or reformation? A paradoxical perspective from management educators. The International Journal of Management Education, 21(2), 100790. https://doi.org/10.1016/j.ijme.2023.100790
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019). RoBERTa: A robustly optimized BERT pretraining approach. arXiv. arXiv:1907.11692. https://doi.org/10.48550/arXiv.1907.11692
Liu, Z., Liu, Q., Chen, J., Huang, S., Tang, J., & Luo, W. (2022). PyKT: A Python Library to Benchmark Deep Learning Based Knowledge Tracing Models. Advances in Neural Information Processing Systems 35, 18542–18555. https://doi.org/10.52202/068431-1347
Liu, Z., Liu, Q., Guo, T., Chen, J., Huang, S., Zhao, X., Tang, J., Luo, W., & Weng, J. (2023). XES3G5M: A knowledge tracing benchmark dataset with auxiliary information. Advances in Neural Information Processing Systems, 36, 32958–32970. https://doi.org/10.52202/075280-1429
Lo, C. K. (2023). What is the impact of ChatGPT on education? A rapid review of the literature. Education Sciences, 13(4), 410. https://doi.org/10.3390/educsci13040410
Ouyang, F., Zheng, L., & Jiao, P. (2022). Artificial intelligence in online higher education: A systematic review of empirical research from 2011 to 2020. Education and Information Technologies, 27(6), 7893–7925. https://doi.org/10.1007/s10639-022-10925-9
Shin, D., Shim, Y., Yu, H., Lee, S., Kim, B., & Choi, Y. (2021). SAINT+: Integrating temporal features for EdNet correctness prediction. In: Proceedings of the 11th International Learning Analytics and Knowledge Conference, April 11-15, 2021, Newport Beach, CA. https://doi.org/10.1145/3448139.3448188
Tlili, A., Shehata, B., Adarkwah, M. A., Bozkurt, A., Hickey, D. T., Huang, R., & Agyemang, B. (2023). What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education. Smart Learning Environments, 10(1), 15. https://doi.org/10.1186/s40561-023-00237-x
Wang, C., Ma, W., Zhang, M., Lv, C., Wan, F., Lin, H., Tang, T., Liu, Y., & Ma, S. (2021). Temporal cross-effects in knowledge tracing. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, (pp. 517–525). Association for Computing Machinery. https://doi.org/10.1145/3437963.3441802
Wu, J., Wang, X., Feng, F., He, X., Chen, L., Lian, J., & Xie, X. (2021). Self-supervised graph learning for recommendation. In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. July 11-15, 2021, Canada. https://doi.org/10.1145/3404835.3462862
Xie, X., Sun, F., Liu, Z., Wu, S., Gao, J., Ding, B., & Cui, B. (2022). Contrastive learning for sequential recommendation. In: Proceedings of the 2022 IEEE 38th International Conference on Data Engineering (ICDE), May 9-12, 2022, Kuala Lumpur, Malaysia https://doi.org/10.1109/ICDE53745.2022.00099
Yu, J., Wang, Y., Zhong, Q., Luo, G., Mao, Y., Sun, K., Feng, W., Xu, W., Cao, S., Zeng, K., et al. (2021). MOOCCubeX: A large knowledge-centered repository for adaptive learning in MOOCs. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management (pp. 4643–4652). Association for Computing Machinery. https://doi.org/10.1145/3459637.3482010
Zhai, X., Chu, X., Chai, C. S., Jong, M. S.-Y., Istenic, A., Spector, M., Liu, J.B., Yuan, J., & Li, Y. (2021). A review of artificial intelligence in education from 2010 to 2020. Complexity, 2021, 8812542. https://doi.org/10.1155/2021/8812542
