Development of an instrument to assess pedagogical utility in e-Learning systems

Evidence bearing upon whether and to what extent learning management systems (LMSs) and similar e-Learning technologies achieve the pedagogical goals for which they were designed has not been well-grounded in theory-based evaluation measures and instruments and sound methodological practice. Usability measures applied to this task typically use technical factors when evaluating the user-system interaction. However, interaction design themes and the centrality of the user’s experience suggests that conventional usability measures are not designed to evaluate the system’s ability to support the goals of the end users in a particular context of use. The purpose of this thesis was to develop the Learning Strategies Support Online (LESSON) questionnaire to assess the utility of e-Learning systems as perceived by the learner along pedagogical rather than technical dimensions. The ability to measure pedagogical support in LMSs provides important opportunities to: (a) improve the system under investigation or similar systems in general, (b) investigate the relationship between intended learning design and actual use of the system and as a result, (c) design online learning environments that optimize the capabilities of LMSs. Key questions guided this research: (1) How might contemporary learning theories be used to design an instrument to measure learning strategy support? (2) What is the factor structure of the questionnaire developed in this thesis? The literature on cognitive and metacognitive strategies associated with self-regulated learning provided the theoretical framework for the initial pool of 117 items. A focus group, expert panel review and pilot provided a collection of evidence bearing upon the validity of interpretations of data generated by the instrument. Finally, the questionnaire was administered to a development sample from which an exploratory factor analysis revealed an 11-item factor structure. Subsequent item reduction based on statistical and conceptual considerations concluded with the refinement of the LESSON to its present form as a 34-item multidimensional scale. Other significant outcomes of this research include (1) a new conceptual framework that advances evaluation measures used in interaction design, (2) the introduction of new theory-based measures, (3) evidence of discrepancies between intended system design and actual use.