Automatic identification of knowledge transforming content in argument essays developed from multiple sources

Developing skills to transform information mined from multiple sources for argumentative writing may help students to articulate convincing evidence for their claims and increase domain knowledge. To successfully engage in knowledge transforming, writers need to strategically select and combine multiple cognitive and metacognitive processes. Many post-secondary students, especially novice writers, struggle to transform knowledge when drawing on multiple sources for essays. External support is needed. As a first step toward developing software that scaffolds knowledge transforming in writing, this study investigated how to identify sentences representing knowledge transformation in argumentative essays. A synthesis of cognitive theories of writing and Bloom’s typology identified 22 linguistic features to model cognitive processes in knowledge transforming, making a methodological contribution to research on multi-source based writing. These features were used as independent variables in a predictive algorithm trained to predict a sentence’s writing mode as knowledge-telling or knowledge-transforming. A corpus of 38 undergraduates’ essays was examined using this algorithm and a coefficient of knowledge transforming was computed for each essay. Two thirds of all evidential sentences were knowledge-telling indicating undergraduates mostly paraphrase or copy information from sources rather than deeply engage with the source material. Eight linguistic features were important predictors of whether an evidential sentence tells or transforms source knowledge: relative position of an evidential sentence in a paragraph, absolute distance between an evidential sentence and the most recent argument, incidence of low- and high-accessibility anaphoric devices, incidence of rhetorical connectives that indicate reasoning, content-word overlap between the evidential sentence and source text, semantic overlap between evidential sentence and preceding/succeeding argument, and semantic overlap between evidential sentence and source text. The machine learning algorithm accurately classified nearly 3 of 4 evidential sentences as knowledge-telling or knowledge-transforming, offering potential for use in future research. The coefficient of knowledge transforming positively but weakly correlated with essay scores assigned by the course instructor. This contrasts with a view that knowledge-telling texts often fail to fulfill writing task requirements.