Assessing the Utility of Deep Learning: Using Learner-System Interaction Data from BioWorld

Peer reviewed: 
Yes, item is peer reviewed.
Scholarly level: 
Final version published as: 

Doleck, T., Poitras, E., & Lajoie, S. (2019). Assessing the Utility of Deep Learning: Using Learner-System Interaction Data from BioWorld. In O. Z.-R. T. B. Johan Van Braak Mark Brown, Lorenzo Cantoni, Manuel Castro, Rhonda Christensen, Gayle V. Davidson-Shivers, Koen DePryck, Martin Ebner, Mikhail Fominykh, Catherine Fulford, Stylianos Hatzipanagos, Gerald Knezek, Karel Kreijns, Gary Marks, Erkko Sointu, Elsebeth Korsgaard Sorensen, Jarmo Viteli, Joke Voogt, Peter Weber, Edgar Weippl (Ed.), Proceedings of EdMedia + Innovate Learning 2019 (pp. 734–738). Association for the Advancement of Computing in Education (AACE).

Date created: 
Machine learning
Deep learning
Educational data mining
Computer-based learning environments
Medical education

In recent years, deep learning (LeCun, Bengio, & Hinton, 2015) has drawn interest in many fields. As optimism for deep learning grows, a better understanding of the efficacy of deep learning is imperative, especially in analyzing and making sense of educational data. This study addresses this issue by establishing a benchmark for a common prediction task – student proficiency in diagnosing patient diseases in a system called BioWorld (Lajoie, 2009). To do so, we compared deep learning to existing solutions, including traditional machine learning algorithms that are commonly used in educational data mining. The dataset consists of log interaction data collected from 30 medical students solving 3 different cases. A 10-fold cross-validation method was used to evaluate the predictive accuracy of each model. Interestingly, our results indicate that deep learning does not outperform traditional machine learning algorithms in predicting diagnosis correctness. We discuss the implications in terms of understanding the proper conditions for its use in educational research.

Document type: 
Rights remain with the authors.