A Multiple Study Investigation of the Evaluation Framework for Learning Analytics: Instrument Validation and the Impact on Learner Performance

Abstract

The purposes of the two studies reported in this research are to adapt and validate the instrument of the Evaluation Framework for Learning Analytics (EFLA) for learners into the Turkish context, and to examine how metacognitive and behavioral factors predict learner performance. Study 1 was conducted with 83 online learners enrolled in a 16-week course delivered through the Moodle learning management system. The findings from the confirmatory factor analysis indicated that a three-factor model of the EFLA for learners provided the best model fit for the collected data. The model is consistent with the factorial structure of the original instrument developed based on the data from the European learners. Study 2 aimed to reveal how the metacognitive and behavioral factors pertaining to the learning analytics dashboard predict learners’ academic performance. A total of 63 online learners enrolled in a 14-week online computing course participated in this study. The results from the logistic regression analysis indicated that online learners more frequently interacted with the learning analytics dashboard demonstrated greater academic performance. Furthermore, the dimensions of the EFLA, together with the interaction with the dashboard, significantly predicted learners’ academic performance. This multiple-study investigation contributes to the generalizability of the EFLA for learners and highlights the importance of metacognitive and behavioral factors for the impact of learning analytics dashboards on learner performance. © 2021, Educational Technology and Society. All Rights Reserved.