Topic driven writing analytics: Using natural language processing techniques to derive topic-based feedback supporting writing revision

Writing provides a medium for learners to construct, critique and share understandings of concepts, reasoning and judgments. Connecting and representing ideas in an essay is an important but challenging skill for novices to develop. Occasions often are limited to receive personalized feedback and time is short to practice implementing recommendations, particularly in post-secondary education. Writing analytics can help address these challenges, providing opportunities to receive on-demand feedback that guides iterative revision to writing. This project creates a new form of writing analytic, analyzing patterns in topics expressed in an individual student’s essay to generate personalized feedback. The analytic is content driven, identifying and describing essay features designed to guide effective revisions, focusing on sequencing topics, expanding underdeveloped ideas, and making holistic revisions to improve the clarity of the ideas expressed. Two experiments tested different reflective prompts based on this analytic, which were derived from unsupervised Latent Dirichlet Allocation (LDA) topic modeling. The analytic visualizes how topics are distributed across an essay. Experiment 1 tested 3 types of feedback encouraging revisions to expand underdeveloped ideas. Model feedback was evaluated by two human reviewers examining 113 undergraduate student essays. The reviewers found feedback prompting sentence-level revisions on underdeveloped topics was most helpful, while analytic feedback on minor topics in the overall essay were not. This indicates a preference for specific prompts in context. Experiment 2 further explored patterns of topic inclusion that could generate personalized feedback at the paragraph level. Prompts were designed for an essay’s introduction and conclusion paragraphs to highlight main topics in the essay potentially overlooked in those paragraphs. Paragraph prompts pointed to revisions to improve topic clarity and cohesion. Model feedback was evaluated using 71 undergraduate student essays scored by two human evaluators. Model accuracy was strong for all types of feedback. This project extends the scope of writing analytics by opening new branches of research on using the content covered in an individual student’s essay to generate novel forms of feedback on concepts covered, connections made, and integration of source materials.