Self-assigned ranking of L2 vocabulary: using the Bricklayer computer game to assess depth of word knowledge

This dissertation concerns vocabulary assessment for second language learners. Current assessment techniques generally focus either on breadth measures to estimate vocabulary size, or depth measures to see how well learners know a few words. A third type of assessment is the checkbox test, which allows learners to self-assess a large number of words; however, the checkbox test is insensitive to partial knowledge and shortcomings with respect to its reliability of results have been identified. A new paradigm is presented and operationalized in the computer program Bricklayer. Bricklayer allows learners to self-assess large numbers of words in a game rather than test format. It validates responses by presenting random quizzes for some of the words. Furthermore, the learners rank word knowledge, providing a mechanism for capturing partial knowledge. In a research study carried out with Bricklayer, 28 intermediate ESL learners were assessed on 72 words. A validity argument established a chain of reasoning to validate the tool. Two post-tests collected evidence for concurrent validity. A standard multiple-choice test for one set of 36 words determined that Bricklayer predicted 61% of known words and 69% of unknown words; results were better for words which were strongly predicted to be known or unknown. A semantic distance test captured incremental knowledge for the other set of 36 words. Bricklayer was sensitive to partial knowledge although its predictive power was weak. Bricklayer’s reliability investigation found 12.5% of the words exhibited poor fit to the predictive model, although there were too few such words to determine a cause. Bricklayer’s performance was found to be comparable to that of the checkbox test. These findings suggest that Bricklayer’s assessment paradigm is a way to build up models of students’ knowledge and behaviour in computer assisted learning environments. Bricklayer provides a first pass in identifying known and unknown words; the remaining words are good candidates for additional assessment or instruction. Several improvements to Bricklayer’s idiosyncrasies were noted. Most importantly, Bricklayer’s scoring represents seven levels of knowledge for each word, although such distinctions of knowledge were not captured. Also, Bricklayer’s forced-choice activity led to results which were highly context-dependent.