A Fully Bayesian Approach to Model Process Data in a Newly Developed Computer-based Assessment

Writing requires substantial conscious effort compared to other literacy skills (e.g., reading). Cognitive models of writing have been proposed to reflect the underlying mechanisms of the writing process. In a computer-based writing assessment, keystroke log data can capture a student’s writing process (e.g., their typing behavior) as the student responds to a writing assignment prompt. Quantification of the writing process may contribute to establishing a writing profile for each student and influence the ongoing writing instruction. Extracting meaningful features from writing process data remains challenging. In this research, for interpretability purposes, we endorse the idea that cognitive models should be the foundation to extract process features. Specifically, we have (1) designed a computer-based assessment to capture students’ writing process; (2) administered the assessment to students to collect response process data; (3) parsed the raw data into pause data; and (4) modeled the pause data with a hierarchical mixture model using a fully Bayesian approach. Under the Bayesian model’s assumption of exchangeability, essays (students) are treated as similar data units, meaning that the model borrows strength from longer essays to enhance the efficiency of estimation for shorter essays. In addition, we provided the source code for designing the keystroke logging system. Therefore, if future researchers intend to develop their own keystroke logging system, they can modify our callback functions, instead of creating the code from scratch.